2024-09-10T20:08:13.989 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T20:08:14.004 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:08:14.090 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest}} email: skanta@redhat.com first_in_suite: false job_id: '7899553' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 3 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) mon_bind_msgr2: false sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZX3/psGZwZdZZ2aosDjxDoMUvnTecZS9rTWj5Xs2Sz2DtkdTTxnyz6ptHA6m3un6d3cVZAZba+kO9rDmcx8BQ= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T20:08:14.090 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T20:08:14.090 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T20:08:14.091 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T20:08:14.092 INFO:teuthology.task.internal:Checking packages... 2024-09-10T20:08:14.114 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T20:08:14.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T20:08:14.114 INFO:teuthology.packaging:ref: None 2024-09-10T20:08:14.114 INFO:teuthology.packaging:tag: None 2024-09-10T20:08:14.115 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:08:14.115 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:08:14.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:08:14.256 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858.gbad1f558 2024-09-10T20:08:14.257 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T20:08:14.263 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T20:08:14.263 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T20:08:14.268 INFO:teuthology.task.internal:Saving configuration 2024-09-10T20:08:14.279 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T20:08:14.287 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T20:08:14.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 20:03:01.286027', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZX3/psGZwZdZZ2aosDjxDoMUvnTecZS9rTWj5Xs2Sz2DtkdTTxnyz6ptHA6m3un6d3cVZAZba+kO9rDmcx8BQ='} 2024-09-10T20:08:14.305 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T20:08:14.311 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-10T20:08:14.311 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T20:08:14.340 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f370fed3be0>, signals=[15]) 2024-09-10T20:08:14.341 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T20:08:14.346 INFO:teuthology.task.internal:Opening connections... 2024-09-10T20:08:14.346 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:08:14.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:08:14.423 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T20:08:14.429 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-09-10T20:08:14.506 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-09-10T20:08:14.507 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T20:08:14.564 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-09-10T20:08:14.565 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-09-10T20:08:14.565 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T20:08:14.565 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-09-10T20:08:14.565 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T20:08:14.565 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T20:08:14.565 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T20:08:14.566 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-09-10T20:08:14.583 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T20:08:14.591 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T20:08:14.665 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T20:08:14.666 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-09-10T20:08:14.682 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T20:08:14.691 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T20:08:14.691 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-09-10T20:08:14.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T20:08:14.964 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T20:08:14.978 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T20:08:14.979 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T20:08:14.979 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T20:08:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:08:14.997 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-10T20:08:15.054 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-503.el9.x86_64 2024-09-10T20:08:15.055 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-503.el9.x86_64 2024-09-10T20:08:15.055 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-09-10T20:08:17.611 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-10T20:08:22.496 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 20 MB 00:03 2024-09-10T20:08:28.047 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-10T20:08:29.522 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-10T20:08:30.726 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-10T20:08:36.174 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-10T20:08:38.230 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T20:08:38.230 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T20:08:38.316 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:08:38.324 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:08:38.324 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-10T20:08:38.324 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:08:38.324 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:08:38.324 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T20:08:38.324 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:08:38.324 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T20:08:38.325 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T20:08:38.325 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T20:08:38.325 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:08:38.325 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:08:38.325 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:08:38.325 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-09-10T20:08:38.325 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:08:38.326 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 79 M 2024-09-10T20:08:38.326 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 126 M 2024-09-10T20:08:38.326 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:08:38.500 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 148 kB/s | 23 kB 00:00 2024-09-10T20:08:39.092 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 46 MB/s | 35 MB 00:00 2024-09-10T20:08:39.259 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-09-10T20:08:39.542 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 27 MB/s | 29 MB 00:01 2024-09-10T20:08:39.543 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:08:39.543 INFO:teuthology.orchestra.run.smithi103.stdout:Total 65 MB/s | 79 MB 00:01 2024-09-10T20:08:40.227 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:08:40.597 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:08:40.597 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:08:41.571 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:08:41.571 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:08:44.050 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:08:44.723 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:08:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:08:47.614 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:08:47.769 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:08:52.579 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:08:52.623 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:08:57.461 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:09:36.232 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:09:36.275 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:09:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:09:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:09:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:09:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:09:37.479 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:09:37.479 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:09:37.479 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:09:37.479 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T20:09:37.480 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T20:09:37.480 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T20:09:37.480 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T20:09:37.480 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:09:37.480 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:09:37.838 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:09:37.921 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:09:37.921 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:09:37.921 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T20:09:37.921 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T20:09:37.921 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T20:09:37.922 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T20:09:37.922 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T20:09:37.922 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:09:37.979 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:09:37.980 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:09:37.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-09-10T20:09:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-09-10T20:09:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-09-10T20:09:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-09-10T20:09:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 82.el9 2024-09-10T20:09:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8274670 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-09-10T20:09:38.018 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:09:38.019 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T20:09:38.019 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T20:09:38.019 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T20:09:38.019 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-09-10T20:09:38.019 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:09:38.019 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T20:09:38.020 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T20:09:38.020 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-505.el9.x86_64 2024-09-10T20:09:38.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T20:09:38.624 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-09-10T20:09:39.964 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T20:09:39.991 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-09-10T20:09:39.997 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-10T20:09:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.hVAoFVZRqo 2024-09-10T20:09:40.013 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hVAoFVZRqo 2024-09-10T20:09:40.079 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.hVAoFVZRqo 2024-09-10T20:09:40.212 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.hVAoFVZRqo is 6KB 2024-09-10T20:09:40.260 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.hVAoFVZRqo 2024-09-10T20:09:40.275 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T20:09:40.337 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T20:09:40.337 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T20:09:40.337 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T20:09:40.337 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T20:09:40.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T20:09:40.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-09-10T20:10:10.521 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T20:10:10.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-10T20:10:10.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:10:29.053 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-10T20:10:38.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-10T20:10:38.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:10:41.117 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-10T20:10:53.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-10T20:10:53.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:11:53.189 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T20:12:08.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-10T20:12:08.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:12:08.614 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-10T20:12:08.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-10T20:12:08.964 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T20:12:08.964 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-10T20:12:08.981 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-505.el9.x86_64 2024-09-10T20:12:08.981 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T20:12:08.981 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T20:12:08.981 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T20:12:09.982 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T20:12:09.983 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T20:12:10.062 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-09-10T20:12:10.083 DEBUG:teuthology.parallel:result is None 2024-09-10T20:12:10.083 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T20:12:10.094 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T20:12:10.094 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T20:12:10.112 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T20:12:10.123 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T20:12:10.137 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T20:12:10.137 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T20:12:10.185 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T20:12:10.196 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T20:12:10.196 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T20:12:10.250 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:10.261 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:10.263 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T20:12:10.271 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T20:12:10.271 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T20:12:10.329 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T20:12:10.340 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T20:12:10.340 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T20:12:10.385 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:12:10.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:12:10.519 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:12:10.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:12:10.643 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:12:10.643 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T20:12:10.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-09-10T20:12:10.792 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:12:10.977 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T20:12:10.986 INFO:teuthology.task.internal:Starting timer... 2024-09-10T20:12:10.986 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T20:12:11.008 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T20:12:11.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-09-10T20:12:11.076 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-09-10T20:12:11.077 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T20:12:11.078 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-09-10T20:12:11.101 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-09-10T20:12:11.101 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-09-10T20:12:11.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:12:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:12:11.166 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T20:12:11.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-09-10T20:12:11.243 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T20:12:11.252 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T20:12:11.264 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T20:12:11.265 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T20:15:50.520 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-09-10T20:15:50.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-10T20:15:50.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:15:50.598 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-10T20:15:50.676 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-10T20:15:50.676 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T20:15:50.689 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T20:15:50.689 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T20:15:50.689 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:15:50.746 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T20:15:50.760 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T20:15:50.785 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-09-10T20:15:50.796 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-09-10T20:15:50.810 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T20:15:50.823 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T20:15:50.879 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:15:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:15:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-10T20:15:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:15:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:15:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:15:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:15:50.883 INFO:teuthology.run_tasks:Running task install... 2024-09-10T20:15:50.891 DEBUG:teuthology.task.install:project ceph 2024-09-10T20:15:50.892 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T20:15:50.892 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T20:15:50.892 INFO:teuthology.task.install:Using flavor: default 2024-09-10T20:15:50.898 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T20:15:50.898 INFO:teuthology.task.install:extra packages: [] 2024-09-10T20:15:50.898 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T20:15:50.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:15:51.048 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T20:15:51.048 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T20:15:51.160 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T20:15:51.161 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:15:51.161 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T20:15:51.186 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T20:15:51.187 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T20:15:51.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T20:15:51.328 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T20:15:51.392 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-09-10T20:15:51.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-09-10T20:15:51.627 INFO:teuthology.orchestra.run.smithi103.stdout:45 files removed 2024-09-10T20:15:51.656 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-09-10T20:15:52.207 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-09-10T20:15:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-10T20:15:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-10T20:15:53.508 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-10T20:15:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-10T20:15:59.970 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-10T20:16:01.587 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-10T20:16:02.817 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-10T20:16:08.265 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-09-10T20:16:10.666 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T20:16:10.692 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T20:16:10.693 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T20:16:10.694 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout:Install 31 Packages 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-09-10T20:16:10.695 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:10.696 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 62 M 2024-09-10T20:16:10.696 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:16:11.491 INFO:teuthology.orchestra.run.smithi103.stdout:(1/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 8.8 MB/s | 5.5 MB 00:00 2024-09-10T20:16:11.541 INFO:teuthology.orchestra.run.smithi103.stdout:(2/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 506 kB/s | 25 kB 00:00 2024-09-10T20:16:11.641 INFO:teuthology.orchestra.run.smithi103.stdout:(3/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-10T20:16:11.809 INFO:teuthology.orchestra.run.smithi103.stdout:(4/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 12 MB/s | 11 MB 00:00 2024-09-10T20:16:11.842 INFO:teuthology.orchestra.run.smithi103.stdout:(5/33): libradosstriper1-19.3.0-4858.gbad1f558. 2.5 MB/s | 509 kB 00:00 2024-09-10T20:16:12.068 INFO:teuthology.orchestra.run.smithi103.stdout:(6/33): ceph-common-19.3.0-4858.gbad1f558.el9.x 18 MB/s | 22 MB 00:01 2024-09-10T20:16:12.093 INFO:teuthology.orchestra.run.smithi103.stdout:(7/33): python3-ceph-argparse-19.3.0-4858.gbad1 180 kB/s | 45 kB 00:00 2024-09-10T20:16:12.118 INFO:teuthology.orchestra.run.smithi103.stdout:(8/33): python3-ceph-common-19.3.0-4858.gbad1f5 2.8 MB/s | 143 kB 00:00 2024-09-10T20:16:12.152 INFO:teuthology.orchestra.run.smithi103.stdout:(9/33): python3-cephfs-19.3.0-4858.gbad1f558.el 2.7 MB/s | 161 kB 00:00 2024-09-10T20:16:12.178 INFO:teuthology.orchestra.run.smithi103.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 5.3 MB/s | 321 kB 00:00 2024-09-10T20:16:12.202 INFO:teuthology.orchestra.run.smithi103.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 5.9 MB/s | 306 kB 00:00 2024-09-10T20:16:12.228 INFO:teuthology.orchestra.run.smithi103.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 1.9 MB/s | 99 kB 00:00 2024-09-10T20:16:12.320 INFO:teuthology.orchestra.run.smithi103.stdout:(13/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_ 11 MB/s | 5.5 MB 00:00 2024-09-10T20:16:12.404 INFO:teuthology.orchestra.run.smithi103.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 533 kB/s | 107 kB 00:00 2024-09-10T20:16:12.454 INFO:teuthology.orchestra.run.smithi103.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-09-10T20:16:12.504 INFO:teuthology.orchestra.run.smithi103.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 618 kB/s | 171 kB 00:00 2024-09-10T20:16:12.532 INFO:teuthology.orchestra.run.smithi103.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 766 kB/s | 160 kB 00:00 2024-09-10T20:16:12.580 INFO:teuthology.orchestra.run.smithi103.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-09-10T20:16:12.605 INFO:teuthology.orchestra.run.smithi103.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-10T20:16:12.638 INFO:teuthology.orchestra.run.smithi103.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-09-10T20:16:12.664 INFO:teuthology.orchestra.run.smithi103.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 612 kB/s | 36 kB 00:00 2024-09-10T20:16:12.689 INFO:teuthology.orchestra.run.smithi103.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-10T20:16:12.715 INFO:teuthology.orchestra.run.smithi103.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 626 kB/s | 47 kB 00:00 2024-09-10T20:16:12.782 INFO:teuthology.orchestra.run.smithi103.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-10T20:16:12.807 INFO:teuthology.orchestra.run.smithi103.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-10T20:16:12.849 INFO:teuthology.orchestra.run.smithi103.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2024-09-10T20:16:12.866 INFO:teuthology.orchestra.run.smithi103.stdout:(27/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 191 kB/s | 26 kB 00:00 2024-09-10T20:16:12.916 INFO:teuthology.orchestra.run.smithi103.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-10T20:16:12.975 INFO:teuthology.orchestra.run.smithi103.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-10T20:16:13.142 INFO:teuthology.orchestra.run.smithi103.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-09-10T20:16:13.209 INFO:teuthology.orchestra.run.smithi103.stdout:(31/33): librados2-19.3.0-4858.gbad1f558.el9.x8 15 MB/s | 3.5 MB 00:00 2024-09-10T20:16:13.301 INFO:teuthology.orchestra.run.smithi103.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.2 MB/s | 4.4 MB 00:00 2024-09-10T20:16:13.426 INFO:teuthology.orchestra.run.smithi103.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-10T20:16:13.431 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:16:13.432 INFO:teuthology.orchestra.run.smithi103.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-10T20:16:13.549 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:16:13.585 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:16:13.586 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:16:14.281 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:16:14.281 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:16:15.818 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:16:15.907 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T20:16:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T20:16:15.965 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T20:16:15.988 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T20:16:16.009 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T20:16:16.199 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T20:16:16.252 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T20:16:16.656 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T20:16:16.709 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T20:16:16.801 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T20:16:16.893 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T20:16:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T20:16:16.999 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T20:16:17.044 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T20:16:17.088 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T20:16:17.158 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T20:16:17.188 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T20:16:17.252 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T20:16:17.287 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T20:16:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T20:16:17.440 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T20:16:17.488 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T20:16:17.528 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T20:16:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T20:16:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T20:16:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T20:16:17.970 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T20:16:18.012 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T20:16:18.104 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T20:16:18.133 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T20:16:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T20:16:18.659 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T20:16:18.815 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T20:16:18.878 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T20:16:18.964 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T20:16:19.000 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T20:16:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T20:16:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T20:16:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T20:16:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T20:16:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T20:16:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T20:16:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T20:16:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T20:16:32.296 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T20:16:32.296 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-09-10T20:16:32.296 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-09-10T20:16:32.296 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-09-10T20:16:32.296 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-09-10T20:16:32.297 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-09-10T20:16:32.297 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-09-10T20:16:32.297 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-09-10T20:16:32.297 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:33.283 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T20:16:33.309 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T20:16:33.309 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:16:33.309 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T20:16:33.309 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:16:33.309 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:16:33.309 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:33.311 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T20:16:33.394 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T20:16:33.394 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T20:16:34.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T20:16:34.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T20:16:34.625 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T20:16:34.625 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T20:16:34.625 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T20:16:34.625 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T20:16:34.625 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T20:16:34.626 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T20:16:34.626 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T20:16:34.626 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T20:16:34.626 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T20:16:35.574 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T20:16:35.574 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:35.574 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-09-10T20:16:35.574 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T20:16:35.575 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T20:16:35.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:35.577 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T20:16:35.577 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T20:16:35.577 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T20:16:35.577 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T20:16:35.578 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T20:16:35.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:35.578 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:16:35.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-09-10T20:16:36.509 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:16:36.651 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:16:36.652 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:36.653 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-09-10T20:16:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:36.653 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 50 M 2024-09-10T20:16:36.653 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 215 M 2024-09-10T20:16:36.653 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:16:36.895 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-10T20:16:36.937 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-10T20:16:36.979 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 709 kB/s | 161 kB 00:00 2024-09-10T20:16:38.296 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-10T20:16:38.298 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:16:38.298 INFO:teuthology.orchestra.run.smithi103.stdout:Total 31 MB/s | 50 MB 00:01 2024-09-10T20:16:38.326 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:16:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:16:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:16:38.722 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:16:38.722 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:16:39.260 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:16:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T20:16:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T20:16:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T20:16:44.129 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T20:16:44.206 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T20:16:46.699 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T20:16:46.700 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T20:16:46.700 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T20:16:46.700 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T20:16:47.049 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T20:16:47.049 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:16:47.050 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:47.050 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:47.050 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T20:16:47.050 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T20:16:47.050 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:16:47.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-09-10T20:16:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:39 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:16:48.047 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:16:48.050 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:48.050 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:16:48.050 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:48.050 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T20:16:48.051 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T20:16:48.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T20:16:48.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T20:16:48.054 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:48.054 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:16:48.054 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:16:48.054 INFO:teuthology.orchestra.run.smithi103.stdout:Install 37 Packages 2024-09-10T20:16:48.054 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:48.054 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 31 M 2024-09-10T20:16:48.054 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 109 M 2024-09-10T20:16:48.055 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:16:48.749 INFO:teuthology.orchestra.run.smithi103.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 56 kB/s | 6.6 kB 00:00 2024-09-10T20:16:48.941 INFO:teuthology.orchestra.run.smithi103.stdout:(2/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-10T20:16:49.000 INFO:teuthology.orchestra.run.smithi103.stdout:(3/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-10T20:16:49.092 INFO:teuthology.orchestra.run.smithi103.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 3.1 MB/s | 291 kB 00:00 2024-09-10T20:16:49.159 INFO:teuthology.orchestra.run.smithi103.stdout:(5/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-10T20:16:49.359 INFO:teuthology.orchestra.run.smithi103.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 949 kB/s | 253 kB 00:00 2024-09-10T20:16:49.536 INFO:teuthology.orchestra.run.smithi103.stdout:(7/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-10T20:16:49.559 INFO:teuthology.orchestra.run.smithi103.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-09-10T20:16:49.610 INFO:teuthology.orchestra.run.smithi103.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-10T20:16:49.660 INFO:teuthology.orchestra.run.smithi103.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.4 MB/s | 1.2 MB 00:00 2024-09-10T20:16:49.685 INFO:teuthology.orchestra.run.smithi103.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-10T20:16:49.719 INFO:teuthology.orchestra.run.smithi103.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-09-10T20:16:49.744 INFO:teuthology.orchestra.run.smithi103.stdout:(13/37): python3-pycparser-2.20-6.el9.noarch.rp 644 kB/s | 135 kB 00:00 2024-09-10T20:16:49.770 INFO:teuthology.orchestra.run.smithi103.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-10T20:16:49.795 INFO:teuthology.orchestra.run.smithi103.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 459 kB/s | 35 kB 00:00 2024-09-10T20:16:49.837 INFO:teuthology.orchestra.run.smithi103.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 836 kB/s | 77 kB 00:00 2024-09-10T20:16:49.862 INFO:teuthology.orchestra.run.smithi103.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 620 kB/s | 42 kB 00:00 2024-09-10T20:16:49.888 INFO:teuthology.orchestra.run.smithi103.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 433 kB/s | 51 kB 00:00 2024-09-10T20:16:49.955 INFO:teuthology.orchestra.run.smithi103.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-09-10T20:16:49.980 INFO:teuthology.orchestra.run.smithi103.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 156 kB/s | 22 kB 00:00 2024-09-10T20:16:50.005 INFO:teuthology.orchestra.run.smithi103.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 370 kB/s | 43 kB 00:00 2024-09-10T20:16:50.039 INFO:teuthology.orchestra.run.smithi103.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-10T20:16:50.064 INFO:teuthology.orchestra.run.smithi103.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 207 kB/s | 11 kB 00:00 2024-09-10T20:16:50.089 INFO:teuthology.orchestra.run.smithi103.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 365 kB/s | 18 kB 00:00 2024-09-10T20:16:50.114 INFO:teuthology.orchestra.run.smithi103.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-10T20:16:50.140 INFO:teuthology.orchestra.run.smithi103.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-10T20:16:50.165 INFO:teuthology.orchestra.run.smithi103.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-09-10T20:16:50.191 INFO:teuthology.orchestra.run.smithi103.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 917 kB/s | 46 kB 00:00 2024-09-10T20:16:50.216 INFO:teuthology.orchestra.run.smithi103.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T20:16:50.241 INFO:teuthology.orchestra.run.smithi103.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-09-10T20:16:50.267 INFO:teuthology.orchestra.run.smithi103.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-10T20:16:50.292 INFO:teuthology.orchestra.run.smithi103.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-10T20:16:50.317 INFO:teuthology.orchestra.run.smithi103.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-10T20:16:50.334 INFO:teuthology.orchestra.run.smithi103.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 532 kB/s | 36 kB 00:00 2024-09-10T20:16:50.351 INFO:teuthology.orchestra.run.smithi103.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-10T20:16:50.460 INFO:teuthology.orchestra.run.smithi103.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-09-10T20:16:50.485 INFO:teuthology.orchestra.run.smithi103.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-09-10T20:16:50.489 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:16:50.489 INFO:teuthology.orchestra.run.smithi103.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-10T20:16:50.637 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:16:50.677 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:16:50.677 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:16:51.160 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:16:51.161 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:16:52.516 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:16:52.572 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T20:16:52.613 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T20:16:52.641 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T20:16:52.712 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T20:16:52.765 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T20:16:52.806 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T20:16:52.902 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T20:16:52.989 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T20:16:53.030 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T20:16:53.073 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T20:16:53.111 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T20:16:53.146 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T20:16:53.180 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T20:16:53.220 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T20:16:53.289 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T20:16:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T20:16:53.371 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T20:16:53.403 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T20:16:53.485 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T20:16:53.631 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T20:16:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T20:16:53.773 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T20:16:53.812 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T20:16:53.861 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T20:16:53.916 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T20:16:54.113 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T20:16:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T20:16:54.236 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T20:16:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T20:16:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T20:16:54.501 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T20:16:54.671 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T20:16:54.701 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T20:16:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T20:16:54.727 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:16:54.727 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T20:16:54.727 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:16:54.727 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:16:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:56.185 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T20:16:56.212 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T20:16:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:16:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T20:16:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:16:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:16:56.213 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:56.677 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T20:16:56.701 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T20:16:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:16:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T20:16:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:16:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:16:56.702 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:56.928 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T20:16:56.952 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T20:16:56.952 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:16:56.952 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T20:16:56.952 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:16:56.952 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:16:56.952 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:57.017 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T20:16:58.839 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T20:16:58.840 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T20:16:58.841 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T20:16:58.843 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T20:16:58.843 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T20:16:58.843 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T20:16:58.843 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T20:16:58.844 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T20:16:58.844 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T20:16:58.844 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T20:16:58.844 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T20:16:58.844 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:59.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T20:16:59.800 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T20:16:59.801 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T20:16:59.802 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T20:16:59.802 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T20:16:59.802 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T20:16:59.802 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T20:16:59.802 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:16:59.802 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:00.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-09-10T20:17:00.769 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:52 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:00.859 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:00.901 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:00.902 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:00.902 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:00.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-09-10T20:17:01.529 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:01.661 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:01.661 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 784 k 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 790 k 2024-09-10T20:17:01.662 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:01.954 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 2.6 MB/s | 784 kB 00:00 2024-09-10T20:17:01.955 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:01.955 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-10T20:17:01.955 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:01.960 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:01.960 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:01.966 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:01.966 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:02.048 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:02.507 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:02.958 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:03.212 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:03.212 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:03.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T20:17:04.021 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:04.154 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:04.155 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:04.155 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-10T20:17:04.155 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:04.155 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:04.155 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 145 k 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 432 k 2024-09-10T20:17:04.156 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:04.370 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 682 kB/s | 145 kB 00:00 2024-09-10T20:17:04.370 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:04.371 INFO:teuthology.orchestra.run.smithi103.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-10T20:17:04.371 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:04.378 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:04.378 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:04.435 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:04.435 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:04.593 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:04.662 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T20:17:04.688 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T20:17:04.688 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:04.688 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T20:17:04.688 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:05.628 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T20:17:05.628 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:05.628 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:05.628 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:05.629 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:05.629 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:05.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-09-10T20:17:06.426 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:58 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:06.521 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:06.563 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:06.564 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:06.564 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:06.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T20:17:07.192 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:07.325 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T20:17:07.327 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-09-10T20:17:07.328 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:07.329 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 19 M 2024-09-10T20:17:07.329 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 159 M 2024-09-10T20:17:07.329 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:07.691 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4858.gba 212 kB/s | 34 kB 00:00 2024-09-10T20:17:07.716 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4858.gbad 88 kB/s | 16 kB 00:00 2024-09-10T20:17:07.941 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-10T20:17:08.017 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-09-10T20:17:08.092 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.1 MB/s | 548 kB 00:00 2024-09-10T20:17:08.117 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-10T20:17:08.201 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-09-10T20:17:08.260 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-09-10T20:17:08.293 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 973 kB/s | 31 kB 00:00 2024-09-10T20:17:08.327 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-09-10T20:17:08.352 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-10T20:17:08.611 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f5 13 MB/s | 14 MB 00:01 2024-09-10T20:17:08.615 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:08.615 INFO:teuthology.orchestra.run.smithi103.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-10T20:17:08.664 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:08.683 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:08.683 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:08.965 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:08.966 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:09.562 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:09.653 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T20:17:09.688 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T20:17:09.735 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T20:17:09.766 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T20:17:09.871 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T20:17:10.001 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T20:17:10.111 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T20:17:10.162 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T20:17:10.191 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T20:17:10.224 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T20:17:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T20:17:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T20:17:15.559 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T20:17:15.559 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T20:17:15.559 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T20:17:15.559 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T20:17:15.559 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T20:17:15.559 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T20:17:15.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T20:17:15.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T20:17:15.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T20:17:15.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T20:17:15.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T20:17:15.560 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T20:17:15.875 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T20:17:15.875 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:15.875 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:15.875 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T20:17:15.875 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:15.875 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:15.876 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:16.124 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T20:17:16.679 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:08 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:16.812 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T20:17:16.813 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-09-10T20:17:16.814 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:16.815 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 38 M 2024-09-10T20:17:16.815 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 212 M 2024-09-10T20:17:16.815 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:17.095 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-10T20:17:17.170 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-10T20:17:17.195 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-09-10T20:17:17.246 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-10T20:17:17.304 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 662 kB/s | 38 kB 00:00 2024-09-10T20:17:17.479 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-48 14 MB/s | 7.4 MB 00:00 2024-09-10T20:17:17.546 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.9 MB/s | 3.0 MB 00:00 2024-09-10T20:17:17.688 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-09-10T20:17:17.780 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 10 MB/s | 4.9 MB 00:00 2024-09-10T20:17:17.814 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 733 kB/s | 245 kB 00:00 2024-09-10T20:17:17.839 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.6 MB/s | 250 kB 00:00 2024-09-10T20:17:18.114 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-10T20:17:18.115 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:18.116 INFO:teuthology.orchestra.run.smithi103.stdout:Total 29 MB/s | 38 MB 00:01 2024-09-10T20:17:18.470 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:18.496 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:18.496 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:18.931 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:18.932 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:19.688 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:19.767 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T20:17:19.813 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T20:17:19.914 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T20:17:19.955 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T20:17:20.148 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T20:17:20.188 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T20:17:20.270 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T20:17:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T20:17:21.044 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T20:17:22.285 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T20:17:23.326 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T20:17:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T20:17:26.065 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T20:17:26.066 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T20:17:26.422 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T20:17:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T20:17:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T20:17:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T20:17:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:26.424 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:26.800 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-09-10T20:17:27.375 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:19 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:27.508 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:27.509 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:27.509 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:17:27.509 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:27.509 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:27.509 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T20:17:27.509 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:17:27.509 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:27.510 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-09-10T20:17:27.511 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:27.511 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.2 M 2024-09-10T20:17:27.511 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-09-10T20:17:27.511 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:28.026 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 290 kB/s | 49 kB 00:00 2024-09-10T20:17:28.126 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 71 kB/s | 19 kB 00:00 2024-09-10T20:17:28.151 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 87 kB/s | 26 kB 00:00 2024-09-10T20:17:28.551 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 423 kB/s | 222 kB 00:00 2024-09-10T20:17:28.577 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 352 kB/s | 159 kB 00:00 2024-09-10T20:17:28.627 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 646 kB/s | 32 kB 00:00 2024-09-10T20:17:28.661 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 421 kB/s | 14 kB 00:00 2024-09-10T20:17:28.694 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 513 kB/s | 279 kB 00:00 2024-09-10T20:17:28.736 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 2.9 MB/s | 220 kB 00:00 2024-09-10T20:17:28.762 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-10T20:17:28.795 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-10T20:17:28.845 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-10T20:17:28.929 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 142 kB/s | 54 kB 00:00 2024-09-10T20:17:28.932 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:28.932 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-09-10T20:17:28.999 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:29.015 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:29.015 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:29.105 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:29.105 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:29.703 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:29.867 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T20:17:29.930 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T20:17:29.986 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T20:17:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T20:17:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T20:17:30.175 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T20:17:30.275 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T20:17:30.333 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T20:17:31.046 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T20:17:31.098 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T20:17:31.132 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T20:17:31.180 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T20:17:31.202 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T20:17:31.979 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T20:17:31.979 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T20:17:31.979 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T20:17:31.979 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T20:17:31.980 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T20:17:32.315 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T20:17:32.315 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:32.315 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:32.316 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:32.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T20:17:32.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T20:17:32.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T20:17:32.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T20:17:32.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T20:17:32.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T20:17:32.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T20:17:32.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T20:17:32.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T20:17:32.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T20:17:32.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T20:17:32.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T20:17:32.318 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:32.318 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:32.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T20:17:33.121 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:25 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:33.254 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:33.254 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:33.255 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:33.256 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-09-10T20:17:33.256 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:33.256 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 7.0 M 2024-09-10T20:17:33.256 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-09-10T20:17:33.256 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:33.633 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 671 kB/s | 154 kB 00:00 2024-09-10T20:17:33.667 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 942 kB/s | 248 kB 00:00 2024-09-10T20:17:33.717 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.2 MB/s | 58 kB 00:00 2024-09-10T20:17:33.751 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.6 MB/s | 548 kB 00:00 2024-09-10T20:17:33.776 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-10T20:17:35.159 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-09-10T20:17:35.161 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:35.162 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.7 MB/s | 7.0 MB 00:01 2024-09-10T20:17:35.232 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:35.243 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:35.243 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:35.342 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:35.342 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:35.648 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:35.828 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T20:17:35.878 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T20:17:36.397 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T20:17:36.481 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T20:17:36.549 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T20:17:36.574 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T20:17:37.384 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T20:17:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T20:17:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T20:17:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T20:17:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T20:17:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T20:17:37.655 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T20:17:37.655 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:37.655 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:37.655 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:37.655 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T20:17:37.656 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T20:17:37.656 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T20:17:37.656 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T20:17:37.656 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T20:17:37.656 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:37.656 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:37.891 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-09-10T20:17:38.456 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:30 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:38.588 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-09-10T20:17:38.590 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:38.590 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 981 k 2024-09-10T20:17:38.590 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.9 M 2024-09-10T20:17:38.590 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:38.729 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 669 kB/s | 80 kB 00:00 2024-09-10T20:17:38.904 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 3.0 MB/s | 901 kB 00:00 2024-09-10T20:17:38.905 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:38.905 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.0 MB/s | 981 kB 00:00 2024-09-10T20:17:38.917 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:38.938 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:38.939 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:39.019 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:39.020 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:39.207 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T20:17:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T20:17:40.119 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T20:17:40.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T20:17:40.400 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T20:17:40.400 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:40.400 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:40.400 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T20:17:40.400 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:40.400 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:40.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-volume 2024-09-10T20:17:41.252 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:33 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:41.390 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:41.391 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:41.391 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-10T20:17:41.391 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:41.391 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:41.391 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T20:17:41.391 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:41.391 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:41.392 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:41.392 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:41.392 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:41.392 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 285 k 2024-09-10T20:17:41.392 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.3 M 2024-09-10T20:17:41.392 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:41.611 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-10T20:17:41.611 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:41.611 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-10T20:17:41.612 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:41.620 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:41.620 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:41.651 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:41.651 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:41.869 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:41.929 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:41.946 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:41.946 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:41.946 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T20:17:41.946 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:43.006 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:43.006 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:43.006 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:43.007 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:43.007 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:43.007 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:43.312 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-09-10T20:17:43.889 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:44.024 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:44.025 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 126 k 2024-09-10T20:17:44.026 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 456 k 2024-09-10T20:17:44.026 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:44.217 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 663 kB/s | 126 kB 00:00 2024-09-10T20:17:44.217 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:44.217 INFO:teuthology.orchestra.run.smithi103.stdout:Total 658 kB/s | 126 kB 00:00 2024-09-10T20:17:44.217 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:44.223 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:44.223 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:44.287 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:44.288 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:44.556 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:45.009 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:45.283 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:45.283 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:45.283 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:45.283 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:45.283 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:45.283 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:45.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-09-10T20:17:46.105 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:46.199 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:46.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-09-10T20:17:46.870 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:47.002 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:47.003 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-10T20:17:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:47.003 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T20:17:47.003 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:47.004 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 32 k 2024-09-10T20:17:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 142 k 2024-09-10T20:17:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:47.180 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 181 kB/s | 32 kB 00:00 2024-09-10T20:17:47.180 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:47.180 INFO:teuthology.orchestra.run.smithi103.stdout:Total 179 kB/s | 32 kB 00:00 2024-09-10T20:17:47.180 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:47.287 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:47.402 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:47.783 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:48.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:48.013 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:48.013 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:48.013 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:48.013 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:48.013 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:48.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-09-10T20:17:48.760 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:48.892 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:48.893 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:48.893 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:48.962 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-09-10T20:17:49.524 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:49.613 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:49.657 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:49.658 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:49.658 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:49.721 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-09-10T20:17:50.280 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:50.370 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:50.412 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:50.413 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:50.413 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:50.482 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-09-10T20:17:51.045 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:51.134 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:51.177 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:51.179 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:51.179 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:51.245 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-09-10T20:17:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:52.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-09-10T20:17:52.565 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:52.655 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:52.698 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:52.699 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-10T20:17:52.699 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:52.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-09-10T20:17:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:53.481 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:53.481 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:53.481 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-10T20:17:53.481 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:53.481 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:53.481 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T20:17:53.481 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:53.482 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:53.482 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:53.482 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:53.482 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:53.482 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 85 k 2024-09-10T20:17:53.482 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 231 k 2024-09-10T20:17:53.482 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:53.676 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 434 kB/s | 85 kB 00:00 2024-09-10T20:17:53.676 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:53.677 INFO:teuthology.orchestra.run.smithi103.stdout:Total 430 kB/s | 85 kB 00:00 2024-09-10T20:17:53.677 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:53.684 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:53.684 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:53.750 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:53.751 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:53.896 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:54.009 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:54.446 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:54.712 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:54.713 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:54.975 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-09-10T20:17:55.544 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:47 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.2 M 2024-09-10T20:17:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 14 M 2024-09-10T20:17:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:56.006 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 9.9 MB/s | 3.2 MB 00:00 2024-09-10T20:17:56.006 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:56.006 INFO:teuthology.orchestra.run.smithi103.stdout:Total 9.9 MB/s | 3.2 MB 00:00 2024-09-10T20:17:56.007 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:56.104 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:56.104 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:56.565 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:56.624 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:56.650 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T20:17:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:17:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:17:56.651 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:57.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:57.796 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:57.796 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:17:57.796 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:57.797 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:57.797 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:17:58.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-09-10T20:17:58.614 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:50 ago on Tue 10 Sep 2024 08:16:08 PM UTC. 2024-09-10T20:17:58.750 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-10T20:17:58.750 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 170 k 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 490 k 2024-09-10T20:17:58.751 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-10T20:17:58.969 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 783 kB/s | 170 kB 00:00 2024-09-10T20:17:58.970 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:58.970 INFO:teuthology.orchestra.run.smithi103.stdout:Total 778 kB/s | 170 kB 00:00 2024-09-10T20:17:58.970 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-10T20:17:58.978 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-10T20:17:58.978 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-10T20:17:59.042 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-10T20:17:59.042 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-10T20:17:59.191 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-10T20:17:59.300 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:17:59.758 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:00.012 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-10T20:18:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:18:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:00.012 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-10T20:18:00.257 DEBUG:teuthology.parallel:result is None 2024-09-10T20:18:00.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:18:00.400 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T20:18:00.421 INFO:teuthology.orchestra.run.smithi103.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T20:18:00.422 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T20:18:00.422 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T20:18:00.424 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T20:18:00.424 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:00.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T20:18:00.487 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T20:18:00.487 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:00.487 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T20:18:00.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T20:18:00.616 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T20:18:00.616 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:00.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T20:18:00.681 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T20:18:00.745 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T20:18:00.746 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:00.746 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T20:18:00.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T20:18:00.875 INFO:teuthology.run_tasks:Running task ceph... 2024-09-10T20:18:00.957 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-10T20:18:00.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-09-10T20:18:00.982 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-10T20:18:00.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T20:18:01.052 INFO:tasks.ceph:Creating extra log directories... 2024-09-10T20:18:01.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T20:18:01.125 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-10T20:18:01.125 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-10T20:18:01.126 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553', 'branch': 'wip-bharath7-testing-2024-09-10-1409', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7899553', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath7-testing-2024-09-10-1409'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'mon_bind_msgr2': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath7-testing-2024-09-10-1409', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 5245, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath7-testing-2024-09-10-1409', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'targets': {'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZX3/psGZwZdZZ2aosDjxDoMUvnTecZS9rTWj5Xs2Sz2DtkdTTxnyz6ptHA6m3un6d3cVZAZba+kO9rDmcx8BQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'mon_bind_msgr2': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_15:30:27', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-10T20:18:01.126 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T20:18:01.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T20:18:01.238 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:01.238 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T20:18:01.294 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T20:18:01.294 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-09-10T20:18:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T20:18:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:18:01.350 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-10T20:18:01.350 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:18:01.350 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:18:01.350 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-10 20:17:56.902809602 +0000 2024-09-10T20:18:01.350 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-10 20:15:31.768355749 +0000 2024-09-10T20:18:01.350 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-10 20:15:31.768355749 +0000 2024-09-10T20:18:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-10 20:15:31.768355749 +0000 2024-09-10T20:18:01.351 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T20:18:01.413 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-10T20:18:01.413 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-10T20:18:01.413 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.00020428 s, 2.5 MB/s 2024-09-10T20:18:01.415 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T20:18:01.471 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-09-10T20:18:01.526 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T20:18:01.526 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:18:01.526 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-10T20:18:01.526 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:18:01.527 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:18:01.527 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-10 20:17:56.902809602 +0000 2024-09-10T20:18:01.527 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-10 20:15:32.102349363 +0000 2024-09-10T20:18:01.527 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-10 20:15:32.102349363 +0000 2024-09-10T20:18:01.527 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-10 20:15:32.102349363 +0000 2024-09-10T20:18:01.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T20:18:01.589 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-10T20:18:01.589 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-10T20:18:01.589 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000173704 s, 2.9 MB/s 2024-09-10T20:18:01.590 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T20:18:01.645 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-09-10T20:18:01.701 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T20:18:01.701 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:18:01.701 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-10T20:18:01.701 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:18:01.701 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:18:01.701 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-10 20:17:56.903809583 +0000 2024-09-10T20:18:01.702 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-10 20:15:32.420343282 +0000 2024-09-10T20:18:01.702 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-10 20:15:32.420343282 +0000 2024-09-10T20:18:01.702 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-10 20:15:32.420343282 +0000 2024-09-10T20:18:01.702 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T20:18:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-10T20:18:01.765 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-10T20:18:01.765 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000232642 s, 2.2 MB/s 2024-09-10T20:18:01.766 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T20:18:01.822 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-10 20:17:56.903809583 +0000 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-10 20:15:32.753336915 +0000 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-10 20:15:32.753336915 +0000 2024-09-10T20:18:01.877 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-10 20:15:32.753336915 +0000 2024-09-10T20:18:01.878 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T20:18:01.940 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-10T20:18:01.940 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-10T20:18:01.940 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000219951 s, 2.3 MB/s 2024-09-10T20:18:01.941 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T20:18:01.998 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:18:01.998 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-10T20:18:01.998 INFO:tasks.ceph:Generating config... 2024-09-10T20:18:01.999 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-10T20:18:01.999 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2024-09-10T20:18:02.000 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-10T20:18:02.000 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2024-09-10T20:18:02.000 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-10T20:18:02.000 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2024-09-10T20:18:02.000 INFO:tasks.ceph:[global] ms type = async 2024-09-10T20:18:02.000 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-10T20:18:02.000 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-10T20:18:02.000 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-10T20:18:02.000 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-10T20:18:02.000 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-10T20:18:02.000 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-10T20:18:02.000 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-10T20:18:02.000 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-10T20:18:02.000 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-10T20:18:02.000 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-10T20:18:02.000 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-10T20:18:02.001 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-10T20:18:02.002 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-10T20:18:02.002 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-10T20:18:02.002 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-10T20:18:02.002 INFO:tasks.ceph:Setting up mon.a... 2024-09-10T20:18:02.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-10T20:18:02.086 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.keyring 2024-09-10T20:18:02.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-10T20:18:02.130 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:18:02.196 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.103:6789]'), ('mon.b', '[v1:172.21.15.103:6790]'), ('mon.c', '[v1:172.21.15.103:6791]')] 2024-09-10T20:18:02.196 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.103:6789],[v1:172.21.15.103:6790],[v1:172.21.15.103:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-10T20:18:02.197 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:02.197 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T20:18:02.251 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.103:6789]' --addv b '[v1:172.21.15.103:6790]' --addv c '[v1:172.21.15.103:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:02.338 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:02.338 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: generated fsid 8fa38b67-7aca-4cf3-911d-119b3351a56d 2024-09-10T20:18:02.338 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = quincy 2024-09-10T20:18:02.338 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 0 2024-09-10T20:18:02.338 INFO:teuthology.orchestra.run.smithi103.stdout:fsid 8fa38b67-7aca-4cf3-911d-119b3351a56d 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:last_changed 2024-09-10T20:18:02.335700+0000 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-09-10T20:18:02.335700+0000 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:min_mon_release 17 (quincy) 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:election_strategy: 3 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:0: v1:172.21.15.103:6789/0 mon.a 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:1: v1:172.21.15.103:6790/0 mon.b 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:2: v1:172.21.15.103:6791/0 mon.c 2024-09-10T20:18:02.339 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-10T20:18:02.340 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T20:18:02.355 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8fa38b67-7aca-4cf3-911d-119b3351a56d... 2024-09-10T20:18:02.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T20:18:02.437 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-09-10T20:18:02.437 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-09-10T20:18:02.437 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T20:18:02.437 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-09-10T20:18:02.437 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.437 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T20:18:02.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T20:18:02.439 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "[v1:172.21.15.103:6789],[v1:172.21.15.103:6790],[v1:172.21.15.103:6791]" 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon client directed command retry = 5 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon client hunt interval max multiple = 2 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon election default strategy = 3 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr beacon grace = 90 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: ms bind msgr2 = False 2024-09-10T20:18:02.440 INFO:teuthology.orchestra.run.smithi103.stdout: ms inject socket failures = 1000 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: ms type = async 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 8fa38b67-7aca-4cf3-911d-119b3351a56d 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.441 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.442 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore compression algorithm = snappy 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore compression mode = aggressive 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore write v2 = False 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore zero block detection = True 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-09-10T20:18:02.443 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify cached snaps = True 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify missing on start = True 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock override recovery settings = True 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: osd op queue = debug_random 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: osd op queue cut off = debug_random 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-09-10T20:18:02.444 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: debug monc = 10 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:18:02.445 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.446 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout: mon scrub interval = 300 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-09-10T20:18:02.447 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-09-10T20:18:02.448 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-10T20:18:02.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-10T20:18:02.543 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-10T20:18:02.543 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:02.543 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-10T20:18:02.559 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:02.559 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-10T20:18:02.614 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:18:02.614 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:02.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T20:18:02.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:18:02.688 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:02.688 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:02.744 INFO:tasks.ceph:Setting up mon nodes... 2024-09-10T20:18:02.744 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-10T20:18:02.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T20:18:02.834 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T20:18:02.836 INFO:tasks.ceph:Setting up mds nodes... 2024-09-10T20:18:02.836 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-10T20:18:02.837 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-10T20:18:02.837 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-10T20:18:02.837 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-10T20:18:02.862 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:18:02.862 INFO:tasks.ceph:role: osd.0 2024-09-10T20:18:02.862 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:18:02.862 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T20:18:02.934 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:18:02.934 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:18:02.935 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:18:03.440 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-10T20:18:03.442 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-10T20:18:03.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-10T20:18:03.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-10T20:18:03.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-10T20:18:03.678 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:18:03.678 INFO:tasks.ceph:role: osd.1 2024-09-10T20:18:03.678 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:18:03.679 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:18:03.752 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:18:04.201 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-10T20:18:04.204 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-10T20:18:04.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-10T20:18:04.297 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-10T20:18:04.327 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-10T20:18:04.392 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:18:04.392 INFO:tasks.ceph:role: osd.2 2024-09-10T20:18:04.392 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:18:04.392 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:18:04.464 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:18:04.949 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-10T20:18:04.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-10T20:18:04.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-10T20:18:05.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-10T20:18:05.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-10T20:18:05.183 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:18:05.184 INFO:tasks.ceph:role: osd.3 2024-09-10T20:18:05.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:18:05.184 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-10T20:18:05.255 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:18:05.256 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:18:05.783 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-10T20:18:05.785 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-10T20:18:05.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-10T20:18:05.941 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-10T20:18:05.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:06.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:06.066+0000 7fa825822640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-10T20:18:06.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:06.066+0000 7fa825822640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-10T20:18:06.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:06.066+0000 7fa825822640 -1 bdev(0x55bb6d8a7800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-10T20:18:06.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:06.066+0000 7fa825822640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-10T20:18:09.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T20:18:09.257 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:09.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:09.344+0000 7f646ed1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-10T20:18:09.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:09.344+0000 7f646ed1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-10T20:18:09.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:09.345+0000 7f646ed1b640 -1 bdev(0x56493faf9800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-10T20:18:09.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:09.345+0000 7f646ed1b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-10T20:18:12.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T20:18:12.287 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:12.374+0000 7f665c1cc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-10T20:18:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:12.375+0000 7f665c1cc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-10T20:18:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:12.375+0000 7f665c1cc640 -1 bdev(0x55ecf0963800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-10T20:18:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:12.375+0000 7f665c1cc640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-10T20:18:15.541 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-10T20:18:15.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:15.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:15.652+0000 7f65c2a2e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-10T20:18:15.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:15.652+0000 7f65c2a2e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-10T20:18:15.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:15.652+0000 7f65c2a2e640 -1 bdev(0x55f7b3477800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-10T20:18:15.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-10T20:18:15.653+0000 7f65c2a2e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-10T20:18:18.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-10T20:18:18.600 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-10T20:18:18.600 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:18.600 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-10T20:18:18.670 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:18.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-10T20:18:18.737 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:18.738 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-10T20:18:18.805 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:18.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-10T20:18:18.871 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:18.871 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-10T20:18:18.937 INFO:tasks.ceph:Adding keys to all mons... 2024-09-10T20:18:18.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T20:18:19.001 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-09-10T20:18:19.001 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQB6qeBmODGUMRAA57Bz3S1QmIWjJO9MZO8dew== 2024-09-10T20:18:19.001 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-09-10T20:18:19.001 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQB+qeBmobQIBBAAM6dF1lnMZGWOTkmAiUSHZA== 2024-09-10T20:18:19.001 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-09-10T20:18:19.001 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCBqeBmLCWiFBAABQKG+L+sZkn83ny6Vt2j0A== 2024-09-10T20:18:19.001 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-09-10T20:18:19.002 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCEqeBmxaltFhAAEWyteyq9JfZFxGEzYncyLA== 2024-09-10T20:18:19.002 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-09-10T20:18:19.002 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCHqeBmMbftJhAAVpowZSQESMnu92kJU1JRMA== 2024-09-10T20:18:19.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T20:18:19.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:18:19.148 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:18:19.200 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:18:19.254 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:18:19.304 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-10T20:18:19.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-10T20:18:19.329 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:18:19.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-10T20:18:19.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-10T20:18:19.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:18:19.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-10T20:18:19.638 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-10T20:18:19.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:18:19.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-10T20:18:19.825 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:18:19.881 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-10T20:18:19.882 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T20:18:19.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T20:18:19.924 INFO:tasks.ceph.mon.a:Started 2024-09-10T20:18:19.924 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T20:18:19.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T20:18:19.926 INFO:tasks.ceph.mon.b:Started 2024-09-10T20:18:19.926 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T20:18:19.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T20:18:19.929 INFO:tasks.ceph.mon.c:Started 2024-09-10T20:18:19.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph config set mgr mgr_pool false --force 2024-09-10T20:18:31.570 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-10T20:18:31.571 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-10T20:18:31.571 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-10T20:18:31.573 INFO:tasks.ceph.mgr.x:Started 2024-09-10T20:18:31.573 DEBUG:tasks.ceph:set 0 configs 2024-09-10T20:18:31.573 INFO:tasks.ceph:Setting crush tunables to default 2024-09-10T20:18:31.573 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd crush tunables default 2024-09-10T20:18:31.753 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2024-09-10T20:18:31.764 INFO:tasks.ceph:check_enable_crimson: False 2024-09-10T20:18:31.764 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-10T20:18:31.764 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:31.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-10T20:18:31.789 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:31.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-10T20:18:31.853 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:31.853 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-10T20:18:31.917 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-10T20:18:31.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-10T20:18:31.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 7f22b5e1-0040-4c13-a24c-ed6610a42665 0 2024-09-10T20:18:32.209 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-09-10T20:18:32.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new dd79e308-70dc-48ff-9e8f-ff6225c2e718 1 2024-09-10T20:18:32.398 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-09-10T20:18:32.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new bca7fcd4-7b60-4f83-a11e-ca382ee1cc34 2 2024-09-10T20:18:32.588 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-09-10T20:18:32.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new a013b06c-c54b-4a97-892b-1b5dad559339 3 2024-09-10T20:18:32.781 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-09-10T20:18:32.791 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-10T20:18:32.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-10T20:18:32.794 INFO:tasks.ceph.osd.0:Started 2024-09-10T20:18:32.794 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-10T20:18:32.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-10T20:18:32.796 INFO:tasks.ceph.osd.1:Started 2024-09-10T20:18:32.796 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-10T20:18:32.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-10T20:18:32.798 INFO:tasks.ceph.osd.2:Started 2024-09-10T20:18:32.798 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-10T20:18:32.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-10T20:18:32.800 INFO:tasks.ceph.osd.3:Started 2024-09-10T20:18:32.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:18:32.993 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:32.993 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":6,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:18:32.773408+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:18:33.004 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-10T20:18:33.004 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-10T20:18:33.406 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:18:33.563 INFO:teuthology.misc.health.smithi103.stdout: 2024-09-10T20:18:33.563 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":6,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:18:32.773408+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:18:33.577 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-10T20:18:34.496 INFO:tasks.ceph.osd.0.smithi103.stderr:2024-09-10T20:18:34.493+0000 7f95bcd23640 -1 Falling back to public interface 2024-09-10T20:18:34.503 INFO:tasks.ceph.osd.2.smithi103.stderr:2024-09-10T20:18:34.500+0000 7f4001812640 -1 Falling back to public interface 2024-09-10T20:18:34.529 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:18:34.526+0000 7fe77b287640 -1 Falling back to public interface 2024-09-10T20:18:34.536 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-09-10T20:18:34.534+0000 7f15f000f640 -1 Falling back to public interface 2024-09-10T20:18:36.874 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-09-10T20:18:36.871+0000 7f45ce8e0640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-10T20:18:36.877 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-09-10T20:18:36.874+0000 7f45ce8e0640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-10T20:18:38.108 INFO:tasks.ceph.osd.0.smithi103.stderr:2024-09-10T20:18:38.105+0000 7f95bcd23640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:18:38.504 INFO:tasks.ceph.osd.2.smithi103.stderr:2024-09-10T20:18:38.501+0000 7f4001812640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:18:38.588 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:18:38.585+0000 7fe77b287640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:18:38.849 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-09-10T20:18:38.846+0000 7f15f000f640 -1 osd.3 0 log_to_monitors true 2024-09-10T20:18:39.888 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:18:39.884+0000 7fe777216640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T20:18:39.889 INFO:tasks.ceph.osd.0.smithi103.stderr:2024-09-10T20:18:39.884+0000 7f95b8cb2640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T20:18:39.889 INFO:tasks.ceph.osd.2.smithi103.stderr:2024-09-10T20:18:39.885+0000 7f3ffd223640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T20:18:39.893 INFO:tasks.ceph.osd.0.smithi103.stderr:2024-09-10T20:18:39.889+0000 7f95b4ad9640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:18:39.894 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:18:39.890+0000 7fe77303d640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:18:39.898 INFO:tasks.ceph.osd.2.smithi103.stderr:2024-09-10T20:18:39.894+0000 7f3ff904a640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:18:39.980 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:18:40.412 INFO:teuthology.misc.health.smithi103.stdout: 2024-09-10T20:18:40.412 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":8,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:18:39.876754+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:18:40.424 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-10T20:18:40.493 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-09-10T20:18:40.489+0000 7f15ec20d640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T20:18:40.497 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-09-10T20:18:40.493+0000 7f15e3818640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:18:46.828 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:18:47.115 INFO:teuthology.misc.health.smithi103.stdout: 2024-09-10T20:18:47.115 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":10,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:18:41.892626+0000","last_up_change":"2024-09-10T20:18:40.887107+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":1573714980}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":1573714980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":1573714980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":1573714980}]},"public_addr":"172.21.15.103:6800/1573714980","cluster_addr":"172.21.15.103:6801/1573714980","heartbeat_back_addr":"172.21.15.103:6803/1573714980","heartbeat_front_addr":"172.21.15.103:6802/1573714980","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":2964732840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":2964732840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":2964732840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":2964732840}]},"public_addr":"172.21.15.103:6808/2964732840","cluster_addr":"172.21.15.103:6809/2964732840","heartbeat_back_addr":"172.21.15.103:6811/2964732840","heartbeat_front_addr":"172.21.15.103:6810/2964732840","state":["exists","up"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3119339934}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":3119339934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":3119339934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":3119339934}]},"public_addr":"172.21.15.103:6812/3119339934","cluster_addr":"172.21.15.103:6813/3119339934","heartbeat_back_addr":"172.21.15.103:6815/3119339934","heartbeat_front_addr":"172.21.15.103:6814/3119339934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:18:47.126 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-10T20:18:47.126 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-10T20:18:47.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-10T20:18:47.127 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-10T20:18:47.408 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:47.421 INFO:teuthology.orchestra.run.smithi103.stdout:[{"version":3,"timestamp":"2024-09-10T20:18:40.491837+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"9110.666704"}]}] 2024-09-10T20:18:47.421 INFO:tasks.ceph_manager:config epoch is 3 2024-09-10T20:18:47.424 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T20:18:47.424 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-10T20:18:47.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-10T20:18:47.753 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":664857878}]},"active_addr":"172.21.15.103:6816/664857878","active_change":"2024-09-10T20:18:35.861337+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1187091145}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2091918066}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3940113537}]}]} 2024-09-10T20:18:47.766 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-10T20:18:47.766 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-10T20:18:47.767 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:18:48.057 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:48.057 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":10,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:18:41.892626+0000","last_up_change":"2024-09-10T20:18:40.887107+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":1573714980}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":1573714980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":1573714980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":1573714980}]},"public_addr":"172.21.15.103:6800/1573714980","cluster_addr":"172.21.15.103:6801/1573714980","heartbeat_back_addr":"172.21.15.103:6803/1573714980","heartbeat_front_addr":"172.21.15.103:6802/1573714980","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":2964732840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":2964732840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":2964732840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":2964732840}]},"public_addr":"172.21.15.103:6808/2964732840","cluster_addr":"172.21.15.103:6809/2964732840","heartbeat_back_addr":"172.21.15.103:6811/2964732840","heartbeat_front_addr":"172.21.15.103:6810/2964732840","state":["exists","up"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3119339934}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":3119339934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":3119339934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":3119339934}]},"public_addr":"172.21.15.103:6812/3119339934","cluster_addr":"172.21.15.103:6813/3119339934","heartbeat_back_addr":"172.21.15.103:6815/3119339934","heartbeat_front_addr":"172.21.15.103:6814/3119339934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:18:48.069 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-10T20:18:48.069 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:18:48.351 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:48.351 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":10,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:18:41.892626+0000","last_up_change":"2024-09-10T20:18:40.887107+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":1573714980}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":1573714980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":1573714980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":1573714980}]},"public_addr":"172.21.15.103:6800/1573714980","cluster_addr":"172.21.15.103:6801/1573714980","heartbeat_back_addr":"172.21.15.103:6803/1573714980","heartbeat_front_addr":"172.21.15.103:6802/1573714980","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":2964732840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":2964732840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":2964732840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":2964732840}]},"public_addr":"172.21.15.103:6808/2964732840","cluster_addr":"172.21.15.103:6809/2964732840","heartbeat_back_addr":"172.21.15.103:6811/2964732840","heartbeat_front_addr":"172.21.15.103:6810/2964732840","state":["exists","up"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3119339934}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":3119339934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":3119339934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":3119339934}]},"public_addr":"172.21.15.103:6812/3119339934","cluster_addr":"172.21.15.103:6813/3119339934","heartbeat_back_addr":"172.21.15.103:6815/3119339934","heartbeat_front_addr":"172.21.15.103:6814/3119339934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:18:48.363 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T20:18:48.363 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-10T20:18:48.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T20:18:48.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T20:18:48.527 INFO:teuthology.orchestra.run.smithi103.stdout:38654705667 2024-09-10T20:18:48.527 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:18:48.529 INFO:teuthology.orchestra.run.smithi103.stdout:38654705668 2024-09-10T20:18:48.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:18:48.530 INFO:teuthology.orchestra.run.smithi103.stdout:38654705667 2024-09-10T20:18:48.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T20:18:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:38654705668 2024-09-10T20:18:48.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:18:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:38654705666 2024-09-10T20:18:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:38654705667 2024-09-10T20:18:48.886 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.1 2024-09-10T20:18:48.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.0 2024-09-10T20:18:48.890 INFO:teuthology.orchestra.run.smithi103.stdout:38654705666 2024-09-10T20:18:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:38654705667 2024-09-10T20:18:48.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.2 2024-09-10T20:18:48.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.3 2024-09-10T20:18:49.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T20:18:49.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:18:49.904 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:18:49.905 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:18:50.206 INFO:teuthology.orchestra.run.smithi103.stdout:38654705668 2024-09-10T20:18:50.207 INFO:teuthology.orchestra.run.smithi103.stdout:38654705667 2024-09-10T20:18:50.221 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.1 2024-09-10T20:18:50.221 DEBUG:teuthology.parallel:result is None 2024-09-10T20:18:50.224 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2024-09-10T20:18:50.224 DEBUG:teuthology.parallel:result is None 2024-09-10T20:18:50.239 INFO:teuthology.orchestra.run.smithi103.stdout:38654705667 2024-09-10T20:18:50.247 INFO:teuthology.orchestra.run.smithi103.stdout:38654705668 2024-09-10T20:18:50.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2024-09-10T20:18:50.252 DEBUG:teuthology.parallel:result is None 2024-09-10T20:18:50.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.3 2024-09-10T20:18:50.260 DEBUG:teuthology.parallel:result is None 2024-09-10T20:18:50.261 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:18:50.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:18:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:50.579 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:18:50.591 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-10T20:18:49.867172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-10T20:18:50.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:18:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:50.869 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:18:50.880 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-10T20:18:49.867172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-10T20:18:50.881 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:18:50.881 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T20:18:50.881 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-10T20:18:50.881 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-10T20:18:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:51.198 INFO:teuthology.orchestra.run.smithi103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T20:18:51.211 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-10T20:18:51.211 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-09-10T20:18:51.239 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:18:51.521 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:51.521 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":10,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:18:41.892626+0000","last_up_change":"2024-09-10T20:18:40.887107+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":1573714980}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":1573714980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":1573714980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":1573714980}]},"public_addr":"172.21.15.103:6800/1573714980","cluster_addr":"172.21.15.103:6801/1573714980","heartbeat_back_addr":"172.21.15.103:6803/1573714980","heartbeat_front_addr":"172.21.15.103:6802/1573714980","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":2964732840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":2964732840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":2964732840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":2964732840}]},"public_addr":"172.21.15.103:6808/2964732840","cluster_addr":"172.21.15.103:6809/2964732840","heartbeat_back_addr":"172.21.15.103:6811/2964732840","heartbeat_front_addr":"172.21.15.103:6810/2964732840","state":["exists","up"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3119339934}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":3119339934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":3119339934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":3119339934}]},"public_addr":"172.21.15.103:6812/3119339934","cluster_addr":"172.21.15.103:6813/3119339934","heartbeat_back_addr":"172.21.15.103:6815/3119339934","heartbeat_front_addr":"172.21.15.103:6814/3119339934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:18:51.533 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-09-10T20:18:51.533 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-10T20:18:51.534 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:18:51.811 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:51.812 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:18:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-10T20:18:49.867172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-10T20:18:51.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:18:52.103 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:18:52.103 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:18:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-10T20:18:51.867577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-10T20:18:52.115 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-10T20:18:52.116 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-09-10T20:18:52.946 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-09-10T20:18:52.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-09-10T20:18:54.941 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_0' created 2024-09-10T20:18:54.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-10T20:18:56.954 INFO:teuthology.orchestra.run.smithi103.stderr:set pool 1 min_size to 2 2024-09-10T20:18:56.987 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-10T20:18:57.970 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-10T20:18:58.986 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-09-10T20:18:58.987 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-09-10T20:18:59.047 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T20:18:59.047 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-10T20:18:59.185 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2024-09-10T20:18:59.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T20:18:59.187 INFO:teuthology.orchestra.run.smithi103.stdout:38654705671 2024-09-10T20:18:59.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:18:59.524 INFO:teuthology.orchestra.run.smithi103.stdout:38654705669 2024-09-10T20:18:59.525 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2024-09-10T20:18:59.539 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705669 for osd.1 2024-09-10T20:18:59.539 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.0 2024-09-10T20:19:00.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T20:19:00.541 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:19:00.845 INFO:teuthology.orchestra.run.smithi103.stdout:38654705671 2024-09-10T20:19:00.847 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2024-09-10T20:19:00.858 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705671 for osd.0 2024-09-10T20:19:00.858 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:00.859 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705670 for osd.1 2024-09-10T20:19:00.859 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:00.860 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-10T20:19:00.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:19:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:01.181 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:19:01.192 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:18:59.868753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108616,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378744,"statfs":{"total":386547056640,"available":386435833856,"internally_reserved":0,"allocated":507904,"data_stored":133834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.950710+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.950710+0000","last_undegraded":"2024-09-10T20:18:55.950710+0000","last_fullsized":"2024-09-10T20:18:55.950710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041508+0000","last_change":"2024-09-10T20:18:56.970070+0000","last_active":"2024-09-10T20:18:59.041508+0000","last_peered":"2024-09-10T20:18:59.041508+0000","last_clean":"2024-09-10T20:18:59.041508+0000","last_became_active":"2024-09-10T20:18:56.969630+0000","last_became_peered":"2024-09-10T20:18:56.969630+0000","last_unstale":"2024-09-10T20:18:59.041508+0000","last_undegraded":"2024-09-10T20:18:59.041508+0000","last_fullsized":"2024-09-10T20:18:59.041508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:24.822874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.946735+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.946735+0000","last_undegraded":"2024-09-10T20:18:55.946735+0000","last_fullsized":"2024-09-10T20:18:55.946735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040773+0000","last_change":"2024-09-10T20:18:56.970901+0000","last_active":"2024-09-10T20:18:59.040773+0000","last_peered":"2024-09-10T20:18:59.040773+0000","last_clean":"2024-09-10T20:18:59.040773+0000","last_became_active":"2024-09-10T20:18:56.970432+0000","last_became_peered":"2024-09-10T20:18:56.970432+0000","last_unstale":"2024-09-10T20:18:59.040773+0000","last_undegraded":"2024-09-10T20:18:59.040773+0000","last_fullsized":"2024-09-10T20:18:59.040773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:16.157114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.951069+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.951069+0000","last_undegraded":"2024-09-10T20:18:55.951069+0000","last_fullsized":"2024-09-10T20:18:55.951069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.951701+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.951701+0000","last_undegraded":"2024-09-10T20:18:55.951701+0000","last_fullsized":"2024-09-10T20:18:55.951701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040747+0000","last_change":"2024-09-10T20:18:56.972029+0000","last_active":"2024-09-10T20:18:59.040747+0000","last_peered":"2024-09-10T20:18:59.040747+0000","last_clean":"2024-09-10T20:18:59.040747+0000","last_became_active":"2024-09-10T20:18:56.971645+0000","last_became_peered":"2024-09-10T20:18:56.971645+0000","last_unstale":"2024-09-10T20:18:59.040747+0000","last_undegraded":"2024-09-10T20:18:59.040747+0000","last_fullsized":"2024-09-10T20:18:59.040747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:36.644202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041448+0000","last_change":"2024-09-10T20:18:56.977006+0000","last_active":"2024-09-10T20:18:59.041448+0000","last_peered":"2024-09-10T20:18:59.041448+0000","last_clean":"2024-09-10T20:18:59.041448+0000","last_became_active":"2024-09-10T20:18:56.976629+0000","last_became_peered":"2024-09-10T20:18:56.976629+0000","last_unstale":"2024-09-10T20:18:59.041448+0000","last_undegraded":"2024-09-10T20:18:59.041448+0000","last_fullsized":"2024-09-10T20:18:59.041448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:57.971877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.952233+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.952233+0000","last_undegraded":"2024-09-10T20:18:55.952233+0000","last_fullsized":"2024-09-10T20:18:55.952233+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.947468+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.947468+0000","last_undegraded":"2024-09-10T20:18:55.947468+0000","last_fullsized":"2024-09-10T20:18:55.947468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.952166+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.952166+0000","last_undegraded":"2024-09-10T20:18:55.952166+0000","last_fullsized":"2024-09-10T20:18:55.952166+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041197+0000","last_change":"2024-09-10T20:18:56.969164+0000","last_active":"2024-09-10T20:18:59.041197+0000","last_peered":"2024-09-10T20:18:59.041197+0000","last_clean":"2024-09-10T20:18:59.041197+0000","last_became_active":"2024-09-10T20:18:56.968786+0000","last_became_peered":"2024-09-10T20:18:56.968786+0000","last_unstale":"2024-09-10T20:18:59.041197+0000","last_undegraded":"2024-09-10T20:18:59.041197+0000","last_fullsized":"2024-09-10T20:18:59.041197+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:04:50.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040548+0000","last_change":"2024-09-10T20:18:56.965645+0000","last_active":"2024-09-10T20:18:59.040548+0000","last_peered":"2024-09-10T20:18:59.040548+0000","last_clean":"2024-09-10T20:18:59.040548+0000","last_became_active":"2024-09-10T20:18:56.965247+0000","last_became_peered":"2024-09-10T20:18:56.965247+0000","last_unstale":"2024-09-10T20:18:59.040548+0000","last_undegraded":"2024-09-10T20:18:59.040548+0000","last_fullsized":"2024-09-10T20:18:59.040548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:05.976082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041081+0000","last_change":"2024-09-10T20:18:56.967697+0000","last_active":"2024-09-10T20:18:59.041081+0000","last_peered":"2024-09-10T20:18:59.041081+0000","last_clean":"2024-09-10T20:18:59.041081+0000","last_became_active":"2024-09-10T20:18:56.967320+0000","last_became_peered":"2024-09-10T20:18:56.967320+0000","last_unstale":"2024-09-10T20:18:59.041081+0000","last_undegraded":"2024-09-10T20:18:59.041081+0000","last_fullsized":"2024-09-10T20:18:59.041081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:44.280643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041369+0000","last_change":"2024-09-10T20:18:56.966175+0000","last_active":"2024-09-10T20:18:59.041369+0000","last_peered":"2024-09-10T20:18:59.041369+0000","last_clean":"2024-09-10T20:18:59.041369+0000","last_became_active":"2024-09-10T20:18:56.965628+0000","last_became_peered":"2024-09-10T20:18:56.965628+0000","last_unstale":"2024-09-10T20:18:59.041369+0000","last_undegraded":"2024-09-10T20:18:59.041369+0000","last_fullsized":"2024-09-10T20:18:59.041369+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:13.026003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041196+0000","last_change":"2024-09-10T20:18:56.976733+0000","last_active":"2024-09-10T20:18:59.041196+0000","last_peered":"2024-09-10T20:18:59.041196+0000","last_clean":"2024-09-10T20:18:59.041196+0000","last_became_active":"2024-09-10T20:18:56.976259+0000","last_became_peered":"2024-09-10T20:18:56.976259+0000","last_unstale":"2024-09-10T20:18:59.041196+0000","last_undegraded":"2024-09-10T20:18:59.041196+0000","last_fullsized":"2024-09-10T20:18:59.041196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:24:59.850999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:19:01.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:19:01.475 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:01.475 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:19:01.486 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:18:59.868753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108616,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378744,"statfs":{"total":386547056640,"available":386435833856,"internally_reserved":0,"allocated":507904,"data_stored":133834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.950710+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.950710+0000","last_undegraded":"2024-09-10T20:18:55.950710+0000","last_fullsized":"2024-09-10T20:18:55.950710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041508+0000","last_change":"2024-09-10T20:18:56.970070+0000","last_active":"2024-09-10T20:18:59.041508+0000","last_peered":"2024-09-10T20:18:59.041508+0000","last_clean":"2024-09-10T20:18:59.041508+0000","last_became_active":"2024-09-10T20:18:56.969630+0000","last_became_peered":"2024-09-10T20:18:56.969630+0000","last_unstale":"2024-09-10T20:18:59.041508+0000","last_undegraded":"2024-09-10T20:18:59.041508+0000","last_fullsized":"2024-09-10T20:18:59.041508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:24.822874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.946735+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.946735+0000","last_undegraded":"2024-09-10T20:18:55.946735+0000","last_fullsized":"2024-09-10T20:18:55.946735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040773+0000","last_change":"2024-09-10T20:18:56.970901+0000","last_active":"2024-09-10T20:18:59.040773+0000","last_peered":"2024-09-10T20:18:59.040773+0000","last_clean":"2024-09-10T20:18:59.040773+0000","last_became_active":"2024-09-10T20:18:56.970432+0000","last_became_peered":"2024-09-10T20:18:56.970432+0000","last_unstale":"2024-09-10T20:18:59.040773+0000","last_undegraded":"2024-09-10T20:18:59.040773+0000","last_fullsized":"2024-09-10T20:18:59.040773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:16.157114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.951069+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.951069+0000","last_undegraded":"2024-09-10T20:18:55.951069+0000","last_fullsized":"2024-09-10T20:18:55.951069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.951701+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.951701+0000","last_undegraded":"2024-09-10T20:18:55.951701+0000","last_fullsized":"2024-09-10T20:18:55.951701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040747+0000","last_change":"2024-09-10T20:18:56.972029+0000","last_active":"2024-09-10T20:18:59.040747+0000","last_peered":"2024-09-10T20:18:59.040747+0000","last_clean":"2024-09-10T20:18:59.040747+0000","last_became_active":"2024-09-10T20:18:56.971645+0000","last_became_peered":"2024-09-10T20:18:56.971645+0000","last_unstale":"2024-09-10T20:18:59.040747+0000","last_undegraded":"2024-09-10T20:18:59.040747+0000","last_fullsized":"2024-09-10T20:18:59.040747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:36.644202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041448+0000","last_change":"2024-09-10T20:18:56.977006+0000","last_active":"2024-09-10T20:18:59.041448+0000","last_peered":"2024-09-10T20:18:59.041448+0000","last_clean":"2024-09-10T20:18:59.041448+0000","last_became_active":"2024-09-10T20:18:56.976629+0000","last_became_peered":"2024-09-10T20:18:56.976629+0000","last_unstale":"2024-09-10T20:18:59.041448+0000","last_undegraded":"2024-09-10T20:18:59.041448+0000","last_fullsized":"2024-09-10T20:18:59.041448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:57.971877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.952233+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.952233+0000","last_undegraded":"2024-09-10T20:18:55.952233+0000","last_fullsized":"2024-09-10T20:18:55.952233+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.947468+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.947468+0000","last_undegraded":"2024-09-10T20:18:55.947468+0000","last_fullsized":"2024-09-10T20:18:55.947468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.952166+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.952166+0000","last_undegraded":"2024-09-10T20:18:55.952166+0000","last_fullsized":"2024-09-10T20:18:55.952166+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041197+0000","last_change":"2024-09-10T20:18:56.969164+0000","last_active":"2024-09-10T20:18:59.041197+0000","last_peered":"2024-09-10T20:18:59.041197+0000","last_clean":"2024-09-10T20:18:59.041197+0000","last_became_active":"2024-09-10T20:18:56.968786+0000","last_became_peered":"2024-09-10T20:18:56.968786+0000","last_unstale":"2024-09-10T20:18:59.041197+0000","last_undegraded":"2024-09-10T20:18:59.041197+0000","last_fullsized":"2024-09-10T20:18:59.041197+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:04:50.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040548+0000","last_change":"2024-09-10T20:18:56.965645+0000","last_active":"2024-09-10T20:18:59.040548+0000","last_peered":"2024-09-10T20:18:59.040548+0000","last_clean":"2024-09-10T20:18:59.040548+0000","last_became_active":"2024-09-10T20:18:56.965247+0000","last_became_peered":"2024-09-10T20:18:56.965247+0000","last_unstale":"2024-09-10T20:18:59.040548+0000","last_undegraded":"2024-09-10T20:18:59.040548+0000","last_fullsized":"2024-09-10T20:18:59.040548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:05.976082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041081+0000","last_change":"2024-09-10T20:18:56.967697+0000","last_active":"2024-09-10T20:18:59.041081+0000","last_peered":"2024-09-10T20:18:59.041081+0000","last_clean":"2024-09-10T20:18:59.041081+0000","last_became_active":"2024-09-10T20:18:56.967320+0000","last_became_peered":"2024-09-10T20:18:56.967320+0000","last_unstale":"2024-09-10T20:18:59.041081+0000","last_undegraded":"2024-09-10T20:18:59.041081+0000","last_fullsized":"2024-09-10T20:18:59.041081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:44.280643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041369+0000","last_change":"2024-09-10T20:18:56.966175+0000","last_active":"2024-09-10T20:18:59.041369+0000","last_peered":"2024-09-10T20:18:59.041369+0000","last_clean":"2024-09-10T20:18:59.041369+0000","last_became_active":"2024-09-10T20:18:56.965628+0000","last_became_peered":"2024-09-10T20:18:56.965628+0000","last_unstale":"2024-09-10T20:18:59.041369+0000","last_undegraded":"2024-09-10T20:18:59.041369+0000","last_fullsized":"2024-09-10T20:18:59.041369+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:13.026003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041196+0000","last_change":"2024-09-10T20:18:56.976733+0000","last_active":"2024-09-10T20:18:59.041196+0000","last_peered":"2024-09-10T20:18:59.041196+0000","last_clean":"2024-09-10T20:18:59.041196+0000","last_became_active":"2024-09-10T20:18:56.976259+0000","last_became_peered":"2024-09-10T20:18:56.976259+0000","last_unstale":"2024-09-10T20:18:59.041196+0000","last_undegraded":"2024-09-10T20:18:59.041196+0000","last_fullsized":"2024-09-10T20:18:59.041196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:24:59.850999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:19:01.487 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:19:01.766 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:01.767 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:19:01.778 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:18:59.868753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108616,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378744,"statfs":{"total":386547056640,"available":386435833856,"internally_reserved":0,"allocated":507904,"data_stored":133834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.950710+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.950710+0000","last_undegraded":"2024-09-10T20:18:55.950710+0000","last_fullsized":"2024-09-10T20:18:55.950710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041508+0000","last_change":"2024-09-10T20:18:56.970070+0000","last_active":"2024-09-10T20:18:59.041508+0000","last_peered":"2024-09-10T20:18:59.041508+0000","last_clean":"2024-09-10T20:18:59.041508+0000","last_became_active":"2024-09-10T20:18:56.969630+0000","last_became_peered":"2024-09-10T20:18:56.969630+0000","last_unstale":"2024-09-10T20:18:59.041508+0000","last_undegraded":"2024-09-10T20:18:59.041508+0000","last_fullsized":"2024-09-10T20:18:59.041508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:24.822874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.946735+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.946735+0000","last_undegraded":"2024-09-10T20:18:55.946735+0000","last_fullsized":"2024-09-10T20:18:55.946735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040773+0000","last_change":"2024-09-10T20:18:56.970901+0000","last_active":"2024-09-10T20:18:59.040773+0000","last_peered":"2024-09-10T20:18:59.040773+0000","last_clean":"2024-09-10T20:18:59.040773+0000","last_became_active":"2024-09-10T20:18:56.970432+0000","last_became_peered":"2024-09-10T20:18:56.970432+0000","last_unstale":"2024-09-10T20:18:59.040773+0000","last_undegraded":"2024-09-10T20:18:59.040773+0000","last_fullsized":"2024-09-10T20:18:59.040773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:16.157114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.951069+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.951069+0000","last_undegraded":"2024-09-10T20:18:55.951069+0000","last_fullsized":"2024-09-10T20:18:55.951069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.951701+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.951701+0000","last_undegraded":"2024-09-10T20:18:55.951701+0000","last_fullsized":"2024-09-10T20:18:55.951701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040747+0000","last_change":"2024-09-10T20:18:56.972029+0000","last_active":"2024-09-10T20:18:59.040747+0000","last_peered":"2024-09-10T20:18:59.040747+0000","last_clean":"2024-09-10T20:18:59.040747+0000","last_became_active":"2024-09-10T20:18:56.971645+0000","last_became_peered":"2024-09-10T20:18:56.971645+0000","last_unstale":"2024-09-10T20:18:59.040747+0000","last_undegraded":"2024-09-10T20:18:59.040747+0000","last_fullsized":"2024-09-10T20:18:59.040747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:36.644202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041448+0000","last_change":"2024-09-10T20:18:56.977006+0000","last_active":"2024-09-10T20:18:59.041448+0000","last_peered":"2024-09-10T20:18:59.041448+0000","last_clean":"2024-09-10T20:18:59.041448+0000","last_became_active":"2024-09-10T20:18:56.976629+0000","last_became_peered":"2024-09-10T20:18:56.976629+0000","last_unstale":"2024-09-10T20:18:59.041448+0000","last_undegraded":"2024-09-10T20:18:59.041448+0000","last_fullsized":"2024-09-10T20:18:59.041448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:57.971877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.952233+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.952233+0000","last_undegraded":"2024-09-10T20:18:55.952233+0000","last_fullsized":"2024-09-10T20:18:55.952233+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.947468+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.947468+0000","last_undegraded":"2024-09-10T20:18:55.947468+0000","last_fullsized":"2024-09-10T20:18:55.947468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-10T20:18:55.952166+0000","last_change":"2024-09-10T20:18:54.933927+0000","last_active":"2024-09-10T20:18:54.933927+0000","last_peered":"2024-09-10T20:18:54.933927+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:18:55.952166+0000","last_undegraded":"2024-09-10T20:18:55.952166+0000","last_fullsized":"2024-09-10T20:18:55.952166+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041197+0000","last_change":"2024-09-10T20:18:56.969164+0000","last_active":"2024-09-10T20:18:59.041197+0000","last_peered":"2024-09-10T20:18:59.041197+0000","last_clean":"2024-09-10T20:18:59.041197+0000","last_became_active":"2024-09-10T20:18:56.968786+0000","last_became_peered":"2024-09-10T20:18:56.968786+0000","last_unstale":"2024-09-10T20:18:59.041197+0000","last_undegraded":"2024-09-10T20:18:59.041197+0000","last_fullsized":"2024-09-10T20:18:59.041197+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:04:50.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040548+0000","last_change":"2024-09-10T20:18:56.965645+0000","last_active":"2024-09-10T20:18:59.040548+0000","last_peered":"2024-09-10T20:18:59.040548+0000","last_clean":"2024-09-10T20:18:59.040548+0000","last_became_active":"2024-09-10T20:18:56.965247+0000","last_became_peered":"2024-09-10T20:18:56.965247+0000","last_unstale":"2024-09-10T20:18:59.040548+0000","last_undegraded":"2024-09-10T20:18:59.040548+0000","last_fullsized":"2024-09-10T20:18:59.040548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:05.976082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041081+0000","last_change":"2024-09-10T20:18:56.967697+0000","last_active":"2024-09-10T20:18:59.041081+0000","last_peered":"2024-09-10T20:18:59.041081+0000","last_clean":"2024-09-10T20:18:59.041081+0000","last_became_active":"2024-09-10T20:18:56.967320+0000","last_became_peered":"2024-09-10T20:18:56.967320+0000","last_unstale":"2024-09-10T20:18:59.041081+0000","last_undegraded":"2024-09-10T20:18:59.041081+0000","last_fullsized":"2024-09-10T20:18:59.041081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:44.280643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041369+0000","last_change":"2024-09-10T20:18:56.966175+0000","last_active":"2024-09-10T20:18:59.041369+0000","last_peered":"2024-09-10T20:18:59.041369+0000","last_clean":"2024-09-10T20:18:59.041369+0000","last_became_active":"2024-09-10T20:18:56.965628+0000","last_became_peered":"2024-09-10T20:18:56.965628+0000","last_unstale":"2024-09-10T20:18:59.041369+0000","last_undegraded":"2024-09-10T20:18:59.041369+0000","last_fullsized":"2024-09-10T20:18:59.041369+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:13.026003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041196+0000","last_change":"2024-09-10T20:18:56.976733+0000","last_active":"2024-09-10T20:18:59.041196+0000","last_peered":"2024-09-10T20:18:59.041196+0000","last_clean":"2024-09-10T20:18:59.041196+0000","last_became_active":"2024-09-10T20:18:56.976259+0000","last_became_peered":"2024-09-10T20:18:56.976259+0000","last_unstale":"2024-09-10T20:18:59.041196+0000","last_undegraded":"2024-09-10T20:18:59.041196+0000","last_fullsized":"2024-09-10T20:18:59.041196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:24:59.850999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:19:04.780 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:19:05.077 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:05.078 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:19:05.089 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-10T20:19:03.869284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108680,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378680,"statfs":{"total":386547056640,"available":386435768320,"internally_reserved":0,"allocated":573440,"data_stored":170576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:57.966288+0000","last_change":"2024-09-10T20:18:56.974018+0000","last_active":"2024-09-10T20:18:57.966288+0000","last_peered":"2024-09-10T20:18:57.966288+0000","last_clean":"2024-09-10T20:18:57.966288+0000","last_became_active":"2024-09-10T20:18:56.973494+0000","last_became_peered":"2024-09-10T20:18:56.973494+0000","last_unstale":"2024-09-10T20:18:57.966288+0000","last_undegraded":"2024-09-10T20:18:57.966288+0000","last_fullsized":"2024-09-10T20:18:57.966288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:14:00.159780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041508+0000","last_change":"2024-09-10T20:18:56.970070+0000","last_active":"2024-09-10T20:18:59.041508+0000","last_peered":"2024-09-10T20:18:59.041508+0000","last_clean":"2024-09-10T20:18:59.041508+0000","last_became_active":"2024-09-10T20:18:56.969630+0000","last_became_peered":"2024-09-10T20:18:56.969630+0000","last_unstale":"2024-09-10T20:18:59.041508+0000","last_undegraded":"2024-09-10T20:18:59.041508+0000","last_fullsized":"2024-09-10T20:18:59.041508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:24.822874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:57.965910+0000","last_change":"2024-09-10T20:18:56.969389+0000","last_active":"2024-09-10T20:18:57.965910+0000","last_peered":"2024-09-10T20:18:57.965910+0000","last_clean":"2024-09-10T20:18:57.965910+0000","last_became_active":"2024-09-10T20:18:56.969001+0000","last_became_peered":"2024-09-10T20:18:56.969001+0000","last_unstale":"2024-09-10T20:18:57.965910+0000","last_undegraded":"2024-09-10T20:18:57.965910+0000","last_fullsized":"2024-09-10T20:18:57.965910+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:52:22.462741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040773+0000","last_change":"2024-09-10T20:18:56.970901+0000","last_active":"2024-09-10T20:18:59.040773+0000","last_peered":"2024-09-10T20:18:59.040773+0000","last_clean":"2024-09-10T20:18:59.040773+0000","last_became_active":"2024-09-10T20:18:56.970432+0000","last_became_peered":"2024-09-10T20:18:56.970432+0000","last_unstale":"2024-09-10T20:18:59.040773+0000","last_undegraded":"2024-09-10T20:18:59.040773+0000","last_fullsized":"2024-09-10T20:18:59.040773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:16.157114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:57.965441+0000","last_change":"2024-09-10T20:18:56.969407+0000","last_active":"2024-09-10T20:18:57.965441+0000","last_peered":"2024-09-10T20:18:57.965441+0000","last_clean":"2024-09-10T20:18:57.965441+0000","last_became_active":"2024-09-10T20:18:56.968977+0000","last_became_peered":"2024-09-10T20:18:56.968977+0000","last_unstale":"2024-09-10T20:18:57.965441+0000","last_undegraded":"2024-09-10T20:18:57.965441+0000","last_fullsized":"2024-09-10T20:18:57.965441+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:31:22.607969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:57.966191+0000","last_change":"2024-09-10T20:18:56.973790+0000","last_active":"2024-09-10T20:18:57.966191+0000","last_peered":"2024-09-10T20:18:57.966191+0000","last_clean":"2024-09-10T20:18:57.966191+0000","last_became_active":"2024-09-10T20:18:56.973386+0000","last_became_peered":"2024-09-10T20:18:56.973386+0000","last_unstale":"2024-09-10T20:18:57.966191+0000","last_undegraded":"2024-09-10T20:18:57.966191+0000","last_fullsized":"2024-09-10T20:18:57.966191+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:58.537894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040747+0000","last_change":"2024-09-10T20:18:56.972029+0000","last_active":"2024-09-10T20:18:59.040747+0000","last_peered":"2024-09-10T20:18:59.040747+0000","last_clean":"2024-09-10T20:18:59.040747+0000","last_became_active":"2024-09-10T20:18:56.971645+0000","last_became_peered":"2024-09-10T20:18:56.971645+0000","last_unstale":"2024-09-10T20:18:59.040747+0000","last_undegraded":"2024-09-10T20:18:59.040747+0000","last_fullsized":"2024-09-10T20:18:59.040747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:36.644202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041448+0000","last_change":"2024-09-10T20:18:56.977006+0000","last_active":"2024-09-10T20:18:59.041448+0000","last_peered":"2024-09-10T20:18:59.041448+0000","last_clean":"2024-09-10T20:18:59.041448+0000","last_became_active":"2024-09-10T20:18:56.976629+0000","last_became_peered":"2024-09-10T20:18:56.976629+0000","last_unstale":"2024-09-10T20:18:59.041448+0000","last_undegraded":"2024-09-10T20:18:59.041448+0000","last_fullsized":"2024-09-10T20:18:59.041448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:57.971877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:57.966359+0000","last_change":"2024-09-10T20:18:56.969293+0000","last_active":"2024-09-10T20:18:57.966359+0000","last_peered":"2024-09-10T20:18:57.966359+0000","last_clean":"2024-09-10T20:18:57.966359+0000","last_became_active":"2024-09-10T20:18:56.968913+0000","last_became_peered":"2024-09-10T20:18:56.968913+0000","last_unstale":"2024-09-10T20:18:57.966359+0000","last_undegraded":"2024-09-10T20:18:57.966359+0000","last_fullsized":"2024-09-10T20:18:57.966359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:26.842255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:57.965928+0000","last_change":"2024-09-10T20:18:56.972532+0000","last_active":"2024-09-10T20:18:57.965928+0000","last_peered":"2024-09-10T20:18:57.965928+0000","last_clean":"2024-09-10T20:18:57.965928+0000","last_became_active":"2024-09-10T20:18:56.972159+0000","last_became_peered":"2024-09-10T20:18:56.972159+0000","last_unstale":"2024-09-10T20:18:57.965928+0000","last_undegraded":"2024-09-10T20:18:57.965928+0000","last_fullsized":"2024-09-10T20:18:57.965928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:29:35.626981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:57.965936+0000","last_change":"2024-09-10T20:18:56.974552+0000","last_active":"2024-09-10T20:18:57.965936+0000","last_peered":"2024-09-10T20:18:57.965936+0000","last_clean":"2024-09-10T20:18:57.965936+0000","last_became_active":"2024-09-10T20:18:56.973412+0000","last_became_peered":"2024-09-10T20:18:56.973412+0000","last_unstale":"2024-09-10T20:18:57.965936+0000","last_undegraded":"2024-09-10T20:18:57.965936+0000","last_fullsized":"2024-09-10T20:18:57.965936+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:16:44.475056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041197+0000","last_change":"2024-09-10T20:18:56.969164+0000","last_active":"2024-09-10T20:18:59.041197+0000","last_peered":"2024-09-10T20:18:59.041197+0000","last_clean":"2024-09-10T20:18:59.041197+0000","last_became_active":"2024-09-10T20:18:56.968786+0000","last_became_peered":"2024-09-10T20:18:56.968786+0000","last_unstale":"2024-09-10T20:18:59.041197+0000","last_undegraded":"2024-09-10T20:18:59.041197+0000","last_fullsized":"2024-09-10T20:18:59.041197+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:04:50.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.040548+0000","last_change":"2024-09-10T20:18:56.965645+0000","last_active":"2024-09-10T20:18:59.040548+0000","last_peered":"2024-09-10T20:18:59.040548+0000","last_clean":"2024-09-10T20:18:59.040548+0000","last_became_active":"2024-09-10T20:18:56.965247+0000","last_became_peered":"2024-09-10T20:18:56.965247+0000","last_unstale":"2024-09-10T20:18:59.040548+0000","last_undegraded":"2024-09-10T20:18:59.040548+0000","last_fullsized":"2024-09-10T20:18:59.040548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:05.976082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041081+0000","last_change":"2024-09-10T20:18:56.967697+0000","last_active":"2024-09-10T20:18:59.041081+0000","last_peered":"2024-09-10T20:18:59.041081+0000","last_clean":"2024-09-10T20:18:59.041081+0000","last_became_active":"2024-09-10T20:18:56.967320+0000","last_became_peered":"2024-09-10T20:18:56.967320+0000","last_unstale":"2024-09-10T20:18:59.041081+0000","last_undegraded":"2024-09-10T20:18:59.041081+0000","last_fullsized":"2024-09-10T20:18:59.041081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:44.280643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041369+0000","last_change":"2024-09-10T20:18:56.966175+0000","last_active":"2024-09-10T20:18:59.041369+0000","last_peered":"2024-09-10T20:18:59.041369+0000","last_clean":"2024-09-10T20:18:59.041369+0000","last_became_active":"2024-09-10T20:18:56.965628+0000","last_became_peered":"2024-09-10T20:18:56.965628+0000","last_unstale":"2024-09-10T20:18:59.041369+0000","last_undegraded":"2024-09-10T20:18:59.041369+0000","last_fullsized":"2024-09-10T20:18:59.041369+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:13.026003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-10T20:18:59.041196+0000","last_change":"2024-09-10T20:18:56.976733+0000","last_active":"2024-09-10T20:18:59.041196+0000","last_peered":"2024-09-10T20:18:59.041196+0000","last_clean":"2024-09-10T20:18:59.041196+0000","last_became_active":"2024-09-10T20:18:56.976259+0000","last_became_peered":"2024-09-10T20:18:56.976259+0000","last_unstale":"2024-09-10T20:18:59.041196+0000","last_undegraded":"2024-09-10T20:18:59.041196+0000","last_fullsized":"2024-09-10T20:18:59.041196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:24:59.850999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:19:05.090 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-10T20:19:05.090 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-10T20:19:05.091 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-10T20:19:05.140 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-10T20:19:05.140 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-10T20:19:05.190 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-09-10T20:19:05.190 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-09-10T20:19:05.238 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-10T20:19:05.238 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-10T20:19:05.287 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-10T20:19:05.287 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-10T20:19:05.337 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-09-10T20:19:05.337 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-09-10T20:19:05.385 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-10T20:19:05.385 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-10T20:19:05.434 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-10T20:19:05.435 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-10T20:19:05.484 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-09-10T20:19:05.485 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-09-10T20:19:05.533 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-10T20:19:05.533 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-10T20:19:05.583 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-10T20:19:05.583 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-10T20:19:05.632 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-09-10T20:19:05.632 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-09-10T20:19:05.680 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-10T20:19:05.680 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-10T20:19:05.730 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-10T20:19:05.730 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-10T20:19:05.779 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-09-10T20:19:05.780 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-09-10T20:19:05.827 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-10T20:19:05.827 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-10T20:19:05.876 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-10T20:19:05.877 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-10T20:19:05.926 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-09-10T20:19:05.926 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-09-10T20:19:05.974 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-10T20:19:05.974 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-10T20:19:06.023 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-10T20:19:06.024 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-10T20:19:06.073 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-09-10T20:19:06.073 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-09-10T20:19:06.121 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-10T20:19:06.121 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-10T20:19:06.170 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-10T20:19:06.170 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-10T20:19:06.219 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-09-10T20:19:06.219 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-09-10T20:19:06.267 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-10T20:19:06.267 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-10T20:19:06.317 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-10T20:19:06.317 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-10T20:19:06.368 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-09-10T20:19:06.369 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-09-10T20:19:06.416 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-10T20:19:06.534 INFO:teuthology.orchestra.run.smithi103.stdout:{} 2024-09-10T20:19:06.535 INFO:teuthology.orchestra.run.smithi103.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-10T20:19:06.545 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-10T20:19:06.545 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:19:06.579 INFO:tasks.ceph.osd.0:Stopped 2024-09-10T20:19:06.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-10T20:19:06.911 INFO:tasks.ceph.mon.a.smithi103.stderr:2024-09-10T20:19:06.908+0000 7f751acc4640 -1 mon.a@0(leader).osd e16 definitely_dead 0 2024-09-10T20:19:06.991 INFO:teuthology.orchestra.run.smithi103.stderr:marked down osd.0. 2024-09-10T20:19:07.010 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-10T20:19:07.010 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:19:07.040 INFO:tasks.ceph.osd.3:Stopped 2024-09-10T20:19:07.040 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-10T20:19:07.372 INFO:tasks.ceph.mon.a.smithi103.stderr:2024-09-10T20:19:07.369+0000 7f751acc4640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-10T20:19:07.996 INFO:teuthology.orchestra.run.smithi103.stderr:marked down osd.3. 2024-09-10T20:19:08.013 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-09-10T20:19:08.013 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-09-10T20:19:09.009 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-10T20:19:09.009 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-10T20:19:09.062 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-10T20:19:09.063 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-10T20:19:09.112 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-09-10T20:19:09.113 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-09-10T20:19:09.161 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-10T20:19:09.161 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-10T20:19:09.210 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-10T20:19:09.211 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-10T20:19:09.261 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-09-10T20:19:09.262 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-09-10T20:19:09.311 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-10T20:19:09.311 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-10T20:19:09.361 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-10T20:19:09.361 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-10T20:19:09.410 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-09-10T20:19:09.410 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-09-10T20:19:09.460 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-10T20:19:09.460 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-10T20:19:09.509 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-10T20:19:09.509 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-10T20:19:09.558 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-09-10T20:19:09.558 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-09-10T20:19:09.607 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-10T20:19:09.607 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-10T20:19:09.656 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-10T20:19:09.656 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-10T20:19:09.705 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-09-10T20:19:09.705 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-09-10T20:19:09.754 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-10T20:19:09.754 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-10T20:19:09.803 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-10T20:19:09.803 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-10T20:19:09.851 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-09-10T20:19:09.852 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-09-10T20:19:09.900 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-10T20:19:09.901 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-10T20:19:09.949 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-10T20:19:09.949 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-10T20:19:09.998 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-09-10T20:19:09.998 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-09-10T20:19:10.048 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-10T20:19:10.048 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-10T20:19:10.096 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-10T20:19:10.097 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-10T20:19:10.146 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-09-10T20:19:10.147 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-09-10T20:19:10.195 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-10T20:19:10.195 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-10T20:19:10.244 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-10T20:19:10.244 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-10T20:19:10.293 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-10T20:19:10.293 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:19:10.317 INFO:tasks.ceph.osd.1:Stopped 2024-09-10T20:19:10.318 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-10T20:19:10.649 INFO:tasks.ceph.mon.a.smithi103.stderr:2024-09-10T20:19:10.646+0000 7f751acc4640 -1 mon.a@0(leader).osd e19 definitely_dead 0 2024-09-10T20:19:11.019 INFO:teuthology.orchestra.run.smithi103.stderr:marked down osd.1. 2024-09-10T20:19:11.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-10T20:19:13.021 INFO:teuthology.orchestra.run.smithi103.stderr:marked osd lost in epoch 20 2024-09-10T20:19:13.039 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-10T20:19:13.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-10T20:19:13.042 INFO:tasks.ceph.osd.0:Started 2024-09-10T20:19:13.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-10T20:19:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-09-10T20:19:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-09-10T20:19:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-09-10T20:19:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:config diff 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:config diff get 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:config get 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:config help [] 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:config set ... 2024-09-10T20:19:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-09-10T20:19:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:19:13.148 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-10T20:19:14.455 INFO:tasks.ceph.osd.0.smithi103.stderr:2024-09-10T20:19:14.452+0000 7f4160c24640 -1 Falling back to public interface 2024-09-10T20:19:18.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-10T20:19:18.239 INFO:tasks.ceph.osd.0.smithi103.stderr:2024-09-10T20:19:18.236+0000 7f4160c24640 -1 osd.0 16 log_to_monitors true 2024-09-10T20:19:18.265 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-09-10T20:19:18.265 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator fragmentation block 2024-09-10T20:19:18.266 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-09-10T20:19:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:19:18.268 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-10T20:19:19.043 INFO:tasks.ceph.osd.0.smithi103.stderr:2024-09-10T20:19:19.040+0000 7f41589da640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:19:23.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-10T20:19:23.385 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-10T20:19:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-10T20:19:23.385 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:23.393 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-09-10T20:19:23.394 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:19:23.675 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:23.676 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":25,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:19:21.034823+0000","last_up_change":"2024-09-10T20:19:20.033475+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-10T20:18:53.939346+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":24,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":3732379198}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":3732379198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":3732379198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":3732379198}]},"public_addr":"172.21.15.103:6800/3732379198","cluster_addr":"172.21.15.103:6801/3732379198","heartbeat_back_addr":"172.21.15.103:6803/3732379198","heartbeat_front_addr":"172.21.15.103:6802/3732379198","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":2964732840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":2964732840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":2964732840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":2964732840}]},"public_addr":"172.21.15.103:6808/2964732840","cluster_addr":"172.21.15.103:6809/2964732840","heartbeat_back_addr":"172.21.15.103:6811/2964732840","heartbeat_front_addr":"172.21.15.103:6810/2964732840","state":["exists"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3119339934}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":3119339934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":3119339934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":3119339934}]},"public_addr":"172.21.15.103:6812/3119339934","cluster_addr":"172.21.15.103:6813/3119339934","heartbeat_back_addr":"172.21.15.103:6815/3119339934","heartbeat_front_addr":"172.21.15.103:6814/3119339934","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:19:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-09-10T20:19:23.688 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-10T20:19:23.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-10T20:19:23.691 INFO:tasks.ceph.osd.3:Started 2024-09-10T20:19:23.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-10T20:19:23.789 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-09-10T20:19:23.789 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-09-10T20:19:23.789 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:config diff 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:config diff get 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:config get 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:config help [] 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:config set ... 2024-09-10T20:19:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-09-10T20:19:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:19:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-10T20:19:24.058 INFO:tasks.ceph.mon.a.smithi103.stderr:2024-09-10T20:19:24.055+0000 7f751d4c9640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 6 pgs recovery_unfound (PG_DAMAGED) 2024-09-10T20:19:25.098 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-09-10T20:19:25.095+0000 7f8693420640 -1 Falling back to public interface 2024-09-10T20:19:28.793 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-10T20:19:28.897 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator fragmentation block 2024-09-10T20:19:28.898 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-09-10T20:19:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:19:28.900 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-10T20:19:28.917 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-09-10T20:19:28.914+0000 7f8693420640 -1 osd.3 17 log_to_monitors true 2024-09-10T20:19:29.131 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-09-10T20:19:29.128+0000 7f8686c2b640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:19:30.012 INFO:tasks.ceph.mon.a.smithi103.stderr:2024-09-10T20:19:30.009+0000 7f751d4c9640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-10T20:19:33.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-10T20:19:34.017 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-10T20:19:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-10T20:19:34.018 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:34.027 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-09-10T20:19:34.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:19:34.312 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:34.313 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":28,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:19:31.135279+0000","last_up_change":"2024-09-10T20:19:30.122960+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-10T20:18:53.939346+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":27,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":3732379198}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":3732379198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":3732379198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":3732379198}]},"public_addr":"172.21.15.103:6800/3732379198","cluster_addr":"172.21.15.103:6801/3732379198","heartbeat_back_addr":"172.21.15.103:6803/3732379198","heartbeat_front_addr":"172.21.15.103:6802/3732379198","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":2964732840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":2964732840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":2964732840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":2964732840}]},"public_addr":"172.21.15.103:6808/2964732840","cluster_addr":"172.21.15.103:6809/2964732840","heartbeat_back_addr":"172.21.15.103:6811/2964732840","heartbeat_front_addr":"172.21.15.103:6810/2964732840","state":["exists"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":27,"up_thru":27,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":3618344502}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":3618344502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":3618344502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":3618344502}]},"public_addr":"172.21.15.103:6808/3618344502","cluster_addr":"172.21.15.103:6809/3618344502","heartbeat_back_addr":"172.21.15.103:6811/3618344502","heartbeat_front_addr":"172.21.15.103:6810/3618344502","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:19:34.325 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-09-10T20:19:34.325 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T20:19:34.325 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T20:19:34.326 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T20:19:34.463 INFO:teuthology.orchestra.run.smithi103.stdout:103079215109 2024-09-10T20:19:34.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:19:34.465 INFO:teuthology.orchestra.run.smithi103.stdout:115964116995 2024-09-10T20:19:34.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:19:34.466 INFO:teuthology.orchestra.run.smithi103.stdout:38654705677 2024-09-10T20:19:34.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:19:34.801 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-09-10T20:19:34.813 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116995 got 1 for osd.3 2024-09-10T20:19:34.823 INFO:teuthology.orchestra.run.smithi103.stdout:103079215107 2024-09-10T20:19:34.826 INFO:teuthology.orchestra.run.smithi103.stdout:38654705676 2024-09-10T20:19:34.837 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215109 got 103079215107 for osd.0 2024-09-10T20:19:34.839 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705676 for osd.2 2024-09-10T20:19:35.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:19:35.838 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:19:35.839 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:19:36.124 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-09-10T20:19:36.137 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116995 got 1 for osd.3 2024-09-10T20:19:36.159 INFO:teuthology.orchestra.run.smithi103.stdout:38654705677 2024-09-10T20:19:36.164 INFO:teuthology.orchestra.run.smithi103.stdout:103079215109 2024-09-10T20:19:36.173 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705677 for osd.2 2024-09-10T20:19:36.173 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:36.177 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215109 got 103079215109 for osd.0 2024-09-10T20:19:36.177 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:37.138 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:19:37.433 INFO:teuthology.orchestra.run.smithi103.stdout:115964116995 2024-09-10T20:19:37.445 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116995 got 115964116995 for osd.3 2024-09-10T20:19:37.446 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:37.446 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-09-10T20:19:37.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:19:37.724 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:37.724 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:19:37.736 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-10T20:19:35.876923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188608,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":110148,"kb_used_data":1068,"kb_used_omap":11,"kb_used_meta":108980,"kb_avail":377377212,"statfs":{"total":386547056640,"available":386434265088,"internally_reserved":0,"allocated":1093632,"data_stored":517248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":111595817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.757321"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":29,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-09-10T20:19:31.153017+0000","last_change":"2024-09-10T20:19:31.152677+0000","last_active":"2024-09-10T20:19:31.153017+0000","last_peered":"2024-09-10T20:19:31.153017+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:19:31.153017+0000","last_undegraded":"2024-09-10T20:19:31.153017+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:00:26.995081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":44,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.160610+0000","last_change":"2024-09-10T20:19:31.160954+0000","last_active":"2024-09-10T20:19:31.160610+0000","last_peered":"2024-09-10T20:19:31.160610+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:19:31.160610+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":57,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.164416+0000","last_change":"2024-09-10T20:19:31.164686+0000","last_active":"2024-09-10T20:19:31.164416+0000","last_peered":"2024-09-10T20:19:31.164416+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:19:31.164416+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":44,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.165080+0000","last_change":"2024-09-10T20:19:31.165346+0000","last_active":"2024-09-10T20:19:31.165080+0000","last_peered":"2024-09-10T20:19:31.165080+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:19:31.165080+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":32,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.165624+0000","last_change":"2024-09-10T20:19:31.166004+0000","last_active":"2024-09-10T20:19:31.165624+0000","last_peered":"2024-09-10T20:19:31.165624+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:19:31.165624+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":29,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-09-10T20:19:31.156388+0000","last_change":"2024-09-10T20:19:31.156062+0000","last_active":"2024-09-10T20:19:31.156388+0000","last_peered":"2024-09-10T20:19:31.156388+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:19:31.156388+0000","last_undegraded":"2024-09-10T20:19:31.156388+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:01.206261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":46,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.159010+0000","last_change":"2024-09-10T20:19:31.159307+0000","last_active":"2024-09-10T20:19:31.159010+0000","last_peered":"2024-09-10T20:19:31.159010+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:19:31.159010+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":45,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.163773+0000","last_change":"2024-09-10T20:19:31.164123+0000","last_active":"2024-09-10T20:19:31.163773+0000","last_peered":"2024-09-10T20:19:31.163773+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:19:31.163773+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":37,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.162770+0000","last_change":"2024-09-10T20:19:31.163150+0000","last_active":"2024-09-10T20:19:31.162770+0000","last_peered":"2024-09-10T20:19:31.162770+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:19:31.162770+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":55,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.157119+0000","last_change":"2024-09-10T20:19:31.157530+0000","last_active":"2024-09-10T20:19:31.157119+0000","last_peered":"2024-09-10T20:19:31.157119+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:19:31.157119+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":33,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.159641+0000","last_change":"2024-09-10T20:19:31.159991+0000","last_active":"2024-09-10T20:19:31.159641+0000","last_peered":"2024-09-10T20:19:31.159641+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:19:31.159641+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":24,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.167642+0000","last_change":"2024-09-10T20:19:31.167642+0000","last_active":"2024-09-10T20:19:31.167642+0000","last_peered":"2024-09-10T20:19:31.167642+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:19:31.167642+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":55,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.166580+0000","last_change":"2024-09-10T20:19:31.166825+0000","last_active":"2024-09-10T20:19:31.166580+0000","last_peered":"2024-09-10T20:19:31.166580+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:19:31.166580+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":32,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.168509+0000","last_change":"2024-09-10T20:19:31.168509+0000","last_active":"2024-09-10T20:19:31.168509+0000","last_peered":"2024-09-10T20:19:31.168509+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:19:31.168509+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":58,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.161502+0000","last_change":"2024-09-10T20:19:31.161788+0000","last_active":"2024-09-10T20:19:31.161502+0000","last_peered":"2024-09-10T20:19:31.161502+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:19:31.161502+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":45,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:31.162146+0000","last_change":"2024-09-10T20:19:31.162447+0000","last_active":"2024-09-10T20:19:31.162146+0000","last_peered":"2024-09-10T20:19:31.162146+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:19:31.162146+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":188608,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964116995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":278528,"data_stored":119972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":28176080},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":352256,"data_stored":193700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":83604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":278528,"data_stored":119972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:19:37.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:19:38.087 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:38.087 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":66,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":3,"osd_up_since":1725999570,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":188608,"bytes_used":112791552,"bytes_avail":386434265088,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-10T20:18:31.256271+0000","services":{}},"progress_events":{}} 2024-09-10T20:19:38.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '8fa38b67-7aca-4cf3-911d-119b3351a56d', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 10, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 66, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 28, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1725999570, 'num_in_osds': 4, 'osd_in_since': 1725999512, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 188608, 'bytes_used': 112791552, 'bytes_avail': 386434265088, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-10T20:18:31:254942+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-10T20:18:31.256271+0000', 'services': {}}, 'progress_events': {}} 2024-09-10T20:19:38.101 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-09-10T20:19:38.101 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T20:19:38.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T20:19:38.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T20:19:38.241 INFO:teuthology.orchestra.run.smithi103.stdout:103079215110 2024-09-10T20:19:38.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:19:38.242 INFO:teuthology.orchestra.run.smithi103.stdout:115964116996 2024-09-10T20:19:38.242 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:19:38.243 INFO:teuthology.orchestra.run.smithi103.stdout:38654705679 2024-09-10T20:19:38.244 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:19:38.582 INFO:teuthology.orchestra.run.smithi103.stdout:115964116995 2024-09-10T20:19:38.587 INFO:teuthology.orchestra.run.smithi103.stdout:38654705678 2024-09-10T20:19:38.588 INFO:teuthology.orchestra.run.smithi103.stdout:103079215109 2024-09-10T20:19:38.597 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116996 got 115964116995 for osd.3 2024-09-10T20:19:38.603 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215110 got 103079215109 for osd.0 2024-09-10T20:19:38.603 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705679 got 38654705678 for osd.2 2024-09-10T20:19:39.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:19:39.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:19:39.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:19:39.911 INFO:teuthology.orchestra.run.smithi103.stdout:115964116995 2024-09-10T20:19:39.924 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116996 got 115964116995 for osd.3 2024-09-10T20:19:39.961 INFO:teuthology.orchestra.run.smithi103.stdout:38654705678 2024-09-10T20:19:39.962 INFO:teuthology.orchestra.run.smithi103.stdout:103079215109 2024-09-10T20:19:39.978 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705679 got 38654705678 for osd.2 2024-09-10T20:19:39.978 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215110 got 103079215109 for osd.0 2024-09-10T20:19:40.925 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:19:40.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:19:40.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:19:41.258 INFO:teuthology.orchestra.run.smithi103.stdout:115964116997 2024-09-10T20:19:41.271 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116996 got 115964116997 for osd.3 2024-09-10T20:19:41.271 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:41.282 INFO:teuthology.orchestra.run.smithi103.stdout:38654705679 2024-09-10T20:19:41.298 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705679 got 38654705679 for osd.2 2024-09-10T20:19:41.298 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:41.305 INFO:teuthology.orchestra.run.smithi103.stdout:103079215111 2024-09-10T20:19:41.318 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215110 got 103079215111 for osd.0 2024-09-10T20:19:41.318 DEBUG:teuthology.parallel:result is None 2024-09-10T20:19:41.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:19:41.704 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:41.705 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":70,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":3,"osd_up_since":1725999570,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":188608,"bytes_used":112791552,"bytes_avail":386434265088,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-10T20:19:37.877677+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:19:41.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '8fa38b67-7aca-4cf3-911d-119b3351a56d', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 10, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 70, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 28, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1725999570, 'num_in_osds': 4, 'osd_in_since': 1725999512, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 188608, 'bytes_used': 112791552, 'bytes_avail': 386434265088, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-10T20:18:31:254942+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-10T20:19:37.877677+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '3': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-10T20:19:41.718 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-09-10T20:19:41.719 DEBUG:teuthology.orchestra.run.smithi103:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-09-10T20:19:41.769 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stderr:INFO: op_size has been rounded to 8192 2024-09-10T20:19:41.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:hints = 1 2024-09-10T20:19:41.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-09-10T20:19:41.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Object prefix: benchmark_data_smithi103_43920 2024-09-10T20:19:41.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:19:41.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 0 0 0 0 0 0 - 0 2024-09-10T20:19:42.769 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 1 20 1328 1308 10.2182 10.2188 0.0165826 0.0151831 2024-09-10T20:19:43.769 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 2 20 2593 2573 10.0501 9.88281 0.0086553 0.0154861 2024-09-10T20:19:44.769 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 3 20 3922 3902 10.1607 10.3828 0.0109707 0.0153423 2024-09-10T20:19:45.769 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 4 20 5195 5175 10.1067 9.94531 0.0119231 0.0154338 2024-09-10T20:19:46.769 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 5 20 6647 6627 10.3538 11.3438 0.00722269 0.014933 2024-09-10T20:19:47.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 6 20 7902 7882 10.2622 9.80469 0.0114303 0.0152076 2024-09-10T20:19:48.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 7 20 9154 9134 10.1934 9.78125 0.00423403 0.0150228 2024-09-10T20:19:49.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 8 20 10175 10155 9.9161 7.97656 0.0130082 0.0157438 2024-09-10T20:19:50.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 9 20 11284 11264 9.77691 8.66406 0.0132332 0.0159633 2024-09-10T20:19:51.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:19:51.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 10 20 12606 12586 9.83194 10.3281 0.011634 0.0158711 2024-09-10T20:19:52.159 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:19:52.160 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:19:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-10T20:19:51.881621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":72417472,"num_objects":8845,"num_object_clones":0,"num_object_copies":35380,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":8899,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":8845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":70858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8872,"ondisk_log_size":8872,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":234648,"kb_used_data":96472,"kb_used_omap":11,"kb_used_meta":138100,"kb_avail":377252712,"statfs":{"total":386547056640,"available":386306777088,"internally_reserved":0,"allocated":98787328,"data_stored":98210944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":141414697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,8,6,6,4,1,0,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":72228864,"num_objects":8817,"num_object_clones":0,"num_object_copies":35268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":70536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003498"},"pg_stats":[{"pgid":"1.f","version":"28'613","reported_seq":642,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:49.079492+0000","last_change":"2024-09-10T20:19:41.780800+0000","last_active":"2024-09-10T20:19:49.079492+0000","last_peered":"2024-09-10T20:19:49.079492+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:19:49.079492+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:00:26.995081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5021696,"num_objects":613,"num_object_clones":0,"num_object_copies":2452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":613,"num_write_kb":4904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":613}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'584","reported_seq":622,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.045191+0000","last_change":"2024-09-10T20:19:31.160954+0000","last_active":"2024-09-10T20:19:49.045191+0000","last_peered":"2024-09-10T20:19:49.045191+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:19:49.045191+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4764832,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":586,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":584,"num_write_kb":4668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":580},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'409","reported_seq":463,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:46.692871+0000","last_change":"2024-09-10T20:19:31.164686+0000","last_active":"2024-09-10T20:19:46.692871+0000","last_peered":"2024-09-10T20:19:46.692871+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:19:46.692871+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3332688,"num_objects":407,"num_object_clones":0,"num_object_copies":1628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":409,"num_write_kb":3262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":406},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'596","reported_seq":636,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.037725+0000","last_change":"2024-09-10T20:19:31.165346+0000","last_active":"2024-09-10T20:19:49.037725+0000","last_peered":"2024-09-10T20:19:49.037725+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:19:49.037725+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4863136,"num_objects":594,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":596,"num_write_kb":4757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":592},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'607","reported_seq":635,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.132710+0000","last_change":"2024-09-10T20:19:31.166004+0000","last_active":"2024-09-10T20:19:49.132710+0000","last_peered":"2024-09-10T20:19:49.132710+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:19:49.132710+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4961440,"num_objects":606,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":610,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":607,"num_write_kb":4853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":604},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'569","reported_seq":598,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:49.077091+0000","last_change":"2024-09-10T20:19:41.823169+0000","last_active":"2024-09-10T20:19:49.077091+0000","last_peered":"2024-09-10T20:19:49.077091+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:19:49.077091+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:01.206261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4661248,"num_objects":569,"num_object_clones":0,"num_object_copies":2276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":569,"num_write_kb":4552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'630","reported_seq":670,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.037210+0000","last_change":"2024-09-10T20:19:31.159307+0000","last_active":"2024-09-10T20:19:49.037210+0000","last_peered":"2024-09-10T20:19:49.037210+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:19:49.037210+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5125280,"num_objects":626,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":630,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":5020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":624},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'620","reported_seq":660,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.016364+0000","last_change":"2024-09-10T20:19:31.164123+0000","last_active":"2024-09-10T20:19:49.016364+0000","last_peered":"2024-09-10T20:19:49.016364+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:19:49.016364+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5051552,"num_objects":617,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":621,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":620,"num_write_kb":4948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":615},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'613","reported_seq":641,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.121781+0000","last_change":"2024-09-10T20:19:31.163150+0000","last_active":"2024-09-10T20:19:49.121781+0000","last_peered":"2024-09-10T20:19:49.121781+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:19:49.121781+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4974912,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":613,"num_write_kb":4881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":605},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'406","reported_seq":459,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:46.677964+0000","last_change":"2024-09-10T20:19:31.157530+0000","last_active":"2024-09-10T20:19:46.677964+0000","last_peered":"2024-09-10T20:19:46.677964+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:19:46.677964+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3323040,"num_objects":406,"num_object_clones":0,"num_object_copies":1624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":410,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":406,"num_write_kb":3246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":404},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'610","reported_seq":638,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.121530+0000","last_change":"2024-09-10T20:19:31.159991+0000","last_active":"2024-09-10T20:19:49.121530+0000","last_peered":"2024-09-10T20:19:49.121530+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:19:49.121530+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4976368,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":610,"num_write_kb":4868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":605},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'631","reported_seq":654,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.133294+0000","last_change":"2024-09-10T20:19:31.167642+0000","last_active":"2024-09-10T20:19:49.133294+0000","last_peered":"2024-09-10T20:19:49.133294+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:19:49.133294+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5167696,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":633,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":631,"num_write_kb":5047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":630},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'414","reported_seq":468,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:46.680381+0000","last_change":"2024-09-10T20:19:31.166825+0000","last_active":"2024-09-10T20:19:46.680381+0000","last_peered":"2024-09-10T20:19:46.680381+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:19:46.680381+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3390032,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":416,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":414,"num_write_kb":3311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":413},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'593","reported_seq":621,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.133724+0000","last_change":"2024-09-10T20:19:31.168509+0000","last_active":"2024-09-10T20:19:49.133724+0000","last_peered":"2024-09-10T20:19:49.133724+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:19:49.133724+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4838560,"num_objects":591,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":595,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":593,"num_write_kb":4740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":589},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'404","reported_seq":457,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:46.697762+0000","last_change":"2024-09-10T20:19:31.161788+0000","last_active":"2024-09-10T20:19:46.697762+0000","last_peered":"2024-09-10T20:19:46.697762+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:19:46.697762+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298464,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":407,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":3229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":401},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'573","reported_seq":613,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-10T20:19:49.018980+0000","last_change":"2024-09-10T20:19:31.162447+0000","last_active":"2024-09-10T20:19:49.018980+0000","last_peered":"2024-09-10T20:19:49.018980+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:19:49.018980+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666528,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":574,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":573,"num_write_kb":4572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":568},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":72417472,"num_objects":8845,"num_object_clones":0,"num_object_copies":35380,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":8899,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":8845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":70858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":105381888,"data_stored":105381888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8872,"ondisk_log_size":8872,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964116999,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76900,"kb_used_data":36884,"kb_used_omap":4,"kb_used_meta":39995,"kb_avail":94294940,"statfs":{"total":96636764160,"available":96558018560,"internally_reserved":0,"allocated":37769216,"data_stored":37610660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":40955600},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,4,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61120,"kb_used_data":25776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94310720,"statfs":{"total":96636764160,"available":96574177280,"internally_reserved":0,"allocated":26394624,"data_stored":26236068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,2,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":83604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69424,"kb_used_data":33632,"kb_used_omap":4,"kb_used_meta":35771,"kb_avail":94302416,"statfs":{"total":96636764160,"available":96565673984,"internally_reserved":0,"allocated":34439168,"data_stored":34280612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":36630223},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38727680,"data_stored":38727680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":27316224,"data_stored":27316224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39297024,"data_stored":39297024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:19:52.172 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-09-10T20:19:52.172 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:52.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_4", 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 1224592654, 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:52.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'3", 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'1", 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_6", 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:52.357 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 3101443614, 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'4", 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'2", 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:52.358 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:52.359 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:52.376 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:52.377 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-09-10T20:19:52.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-09-10T20:19:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:52.638 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:52.638 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-09-10T20:19:52.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 11 20 13498 13478 9.57161 6.96875 0.0147613 0.0163046 2024-09-10T20:19:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:52.855 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 1, 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 1, 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_7", 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 2358579165, 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:52.856 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'3", 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "2(0)" 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:52.857 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(1)", 2024-09-10T20:19:52.858 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:52.858 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:52.858 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:52.858 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:52.877 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-10T20:19:52.877 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-09-10T20:19:52.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-09-10T20:19:53.113 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-10T20:19:53.133 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:53.133 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-09-10T20:19:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:53.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_5", 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 4248945696, 2024-09-10T20:19:53.253 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'3", 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:53.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_5", 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 4212789968, 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'4", 2024-09-10T20:19:53.255 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:53.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:53.257 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:53.257 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:53.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:53.257 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:53.257 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:53.261 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:53.262 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-09-10T20:19:53.262 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-09-10T20:19:53.457 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:53.477 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:53.478 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-09-10T20:19:53.694 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_3", 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 459768081, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'3", 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'1", 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_3", 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 480221105, 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'2", 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(1)", 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:53.699 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:53.716 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-10T20:19:53.716 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-09-10T20:19:53.716 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-09-10T20:19:53.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 12 20 14431 14411 9.38135 7.28906 0.0106532 0.0166373 2024-09-10T20:19:53.984 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:54.005 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-09-10T20:19:54.005 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:54.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-09-10T20:19:54.227 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:54.228 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_6", 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 1708673474, 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'5", 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:54.229 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_9", 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 831402034, 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:54.230 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'6", 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:54.231 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(2)", 2024-09-10T20:19:54.232 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:54.232 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:54.232 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:54.232 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:54.232 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:54.248 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-10T20:19:54.248 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-09-10T20:19:54.249 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-09-10T20:19:54.499 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:54.518 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:54.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-09-10T20:19:54.744 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_8", 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 1901787523, 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'5", 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'3", 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_1", 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 3122379363, 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'4", 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:54.749 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:54.768 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:54.768 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-09-10T20:19:54.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-09-10T20:19:54.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 13 20 15548 15528 9.33092 8.72656 0.0152866 0.0167309 2024-09-10T20:19:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:55.038 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:55.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-09-10T20:19:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.257 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 3, 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 3, 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_4", 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 873891716, 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'8", 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_7", 2024-09-10T20:19:55.259 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:55.260 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:55.260 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 3996155460, 2024-09-10T20:19:55.260 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:55.260 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:55.260 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:55.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.260 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'9", 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'6", 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_2", 2024-09-10T20:19:55.261 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 1767667508, 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'7", 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:55.262 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(3)", 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:55.263 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:55.264 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:55.274 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-10T20:19:55.274 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-09-10T20:19:55.275 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-09-10T20:19:55.516 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-10T20:19:55.537 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:55.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-09-10T20:19:55.763 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:55.763 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:55.763 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:55.763 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:55.763 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_1", 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:55.764 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 26281365, 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'1", 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: "2(0)" 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:55.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_2", 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 3196340981, 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'2", 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:55.766 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: "2(0)" 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(3)", 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:55.767 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:55.768 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:55.768 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:55.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 14 20 16642 16622 9.27487 8.54688 0.021449 0.0168339 2024-09-10T20:19:55.787 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-10T20:19:55.787 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-09-10T20:19:55.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-09-10T20:19:56.032 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:56.050 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:56.051 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-09-10T20:19:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 3, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 3, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_8", 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 1861424966, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'5", 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:56.285 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_6", 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 3910131350, 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'4", 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:56.286 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_4", 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 2787839990, 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'3", 2024-09-10T20:19:56.287 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "2(3)" 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(2)", 2024-09-10T20:19:56.288 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:56.289 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:56.289 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:56.289 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:56.293 INFO:tasks.ceph.mon.a.smithi103.stderr:2024-09-10T20:19:56.287+0000 7f751d4c9640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 11 pgs recovery_unfound (PG_DAMAGED) 2024-09-10T20:19:56.307 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-10T20:19:56.307 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-09-10T20:19:56.307 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-09-10T20:19:56.558 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-10T20:19:56.577 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:56.577 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-09-10T20:19:56.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 15 20 17710 17690 9.21275 8.34375 0.0194384 0.016942 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:56.791 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 1, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 1, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_7", 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 3761017527, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'1", 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:56.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:56.794 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:56.794 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:56.794 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:56.794 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:56.794 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:56.814 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:56.814 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-09-10T20:19:56.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-09-10T20:19:57.059 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-10T20:19:57.082 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:57.082 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-09-10T20:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 1, 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 1, 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_8", 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 2730327960, 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'1", 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "2(1)" 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:57.324 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:57.324 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-09-10T20:19:57.325 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-09-10T20:19:57.568 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-10T20:19:57.591 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:57.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-09-10T20:19:57.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 16 20 18769 18749 9.154 8.27344 0.0155801 0.0170533 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:57.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_5", 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 2998549577, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:57.817 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'4", 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'2", 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_2", 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 4032093161, 2024-09-10T20:19:57.818 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'3", 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'1", 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:57.819 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:57.820 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:57.842 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:57.842 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-09-10T20:19:57.842 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-09-10T20:19:58.082 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:58.105 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:58.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-09-10T20:19:58.344 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:58.344 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:58.344 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:58.344 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:58.344 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:58.344 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:58.344 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_9", 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 1217469130, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:58.345 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'3", 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'1", 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "2(1)" 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "new_9", 2024-09-10T20:19:58.346 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 170713514, 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'2", 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: "2(1)" 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:58.347 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:58.348 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:58.365 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:58.365 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-09-10T20:19:58.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-09-10T20:19:58.611 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:58.635 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-09-10T20:19:58.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-09-10T20:19:58.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 17 20 19813 19793 9.0952 8.15625 0.0171042 0.017164 2024-09-10T20:19:58.862 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:19:58.862 INFO:teuthology.orchestra.run.smithi103.stdout: "offset": { 2024-09-10T20:19:58.862 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "", 2024-09-10T20:19:58.862 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:58.862 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": 0, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 0, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": -9223372036854775808, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "num_missing": 2, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "num_unfound": 2, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "objects": [ 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existed_3", 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:58.863 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 943960171, 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'5", 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "0'0", 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:58.864 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": { 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "oid": "existing_1", 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "snapid": -2, 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "hash": 2885884443, 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 0, 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "namespace": "" 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "need": "19'4", 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "have": "16'1", 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "none", 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "locations": [ 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: "2(2)" 2024-09-10T20:19:58.865 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "NotRecovering", 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: "available_might_have_unfound": true, 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: "might_have_unfound": [ 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": "1(0)", 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "osd is down" 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-09-10T20:19:58.866 INFO:teuthology.orchestra.run.smithi103.stdout: "more": false 2024-09-10T20:19:58.867 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:19:58.890 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-10T20:19:58.890 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-09-10T20:19:58.890 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-09-10T20:19:59.174 INFO:teuthology.orchestra.run.smithi103.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-10T20:19:59.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-10T20:19:59.386 INFO:teuthology.orchestra.run.smithi103.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-10T20:19:59.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-10T20:19:59.514 INFO:teuthology.orchestra.run.smithi103.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-10T20:19:59.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-09-10T20:19:59.757 INFO:teuthology.orchestra.run.smithi103.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-10T20:19:59.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 18 20 20479 20459 8.87895 5.20312 0.0183614 0.017585 2024-09-10T20:19:59.780 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T20:19:59.780 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T20:19:59.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T20:20:00.002 INFO:tasks.ceph.mon.a.smithi103.stderr:2024-09-10T20:19:59.999+0000 7f751d4c9640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 18/15640 objects unfound (0.115%); 1 osds down; Possible data damage: 9 pgs recovery_unfound; Degraded data redundancy: 15676/62560 objects degraded (25.058%), 16 pgs degraded 2024-09-10T20:20:00.033 INFO:teuthology.orchestra.run.smithi103.stdout:115964117002 2024-09-10T20:20:00.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:20:00.059 INFO:teuthology.orchestra.run.smithi103.stdout:103079215116 2024-09-10T20:20:00.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:20:00.067 INFO:teuthology.orchestra.run.smithi103.stdout:38654705684 2024-09-10T20:20:00.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:20:00.576 INFO:teuthology.orchestra.run.smithi103.stdout:38654705683 2024-09-10T20:20:00.577 INFO:teuthology.orchestra.run.smithi103.stdout:115964117001 2024-09-10T20:20:00.580 INFO:teuthology.orchestra.run.smithi103.stdout:103079215115 2024-09-10T20:20:00.590 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705684 got 38654705683 for osd.2 2024-09-10T20:20:00.591 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964117002 got 115964117001 for osd.3 2024-09-10T20:20:00.594 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215116 got 103079215115 for osd.0 2024-09-10T20:20:00.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 19 20 21110 21090 8.67103 4.92969 0.0208006 0.0180049 2024-09-10T20:20:01.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:20:01.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:20:01.595 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:20:01.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:20:01.768922+0000 min lat: 0.00230373 max lat: 0.240861 avg lat: 0.0182044 2024-09-10T20:20:01.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:20:01.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 20 20 21976 21956 8.5757 6.76562 0.0172385 0.0182044 2024-09-10T20:20:02.160 INFO:teuthology.orchestra.run.smithi103.stdout:115964117001 2024-09-10T20:20:02.186 INFO:teuthology.orchestra.run.smithi103.stdout:38654705683 2024-09-10T20:20:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964117002 got 115964117001 for osd.3 2024-09-10T20:20:02.210 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705684 got 38654705683 for osd.2 2024-09-10T20:20:02.212 INFO:teuthology.orchestra.run.smithi103.stdout:103079215115 2024-09-10T20:20:02.241 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215116 got 103079215115 for osd.0 2024-09-10T20:20:02.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 21 20 22705 22685 8.43851 5.69531 0.210388 0.0183998 2024-09-10T20:20:03.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:20:03.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:20:03.242 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:20:03.714 INFO:teuthology.orchestra.run.smithi103.stdout:115964117002 2024-09-10T20:20:03.738 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964117002 got 115964117002 for osd.3 2024-09-10T20:20:03.738 DEBUG:teuthology.parallel:result is None 2024-09-10T20:20:03.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 22 20 23434 23414 8.3138 5.69531 0.0215282 0.0187819 2024-09-10T20:20:03.797 INFO:teuthology.orchestra.run.smithi103.stdout:38654705684 2024-09-10T20:20:03.820 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705684 got 38654705684 for osd.2 2024-09-10T20:20:03.820 DEBUG:teuthology.parallel:result is None 2024-09-10T20:20:03.886 INFO:teuthology.orchestra.run.smithi103.stdout:103079215116 2024-09-10T20:20:03.911 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215116 got 103079215116 for osd.0 2024-09-10T20:20:03.911 DEBUG:teuthology.parallel:result is None 2024-09-10T20:20:03.911 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-10T20:20:03.911 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:04.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:04.292 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:04.319 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-10T20:20:03.884961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":172179648,"num_objects":21023,"num_object_clones":0,"num_object_copies":84092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21050,"num_write_kb":168282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21077,"ondisk_log_size":21077,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":501372,"kb_used_data":250748,"kb_used_omap":11,"kb_used_meta":250548,"kb_avail":376985988,"statfs":{"total":386547056640,"available":386033651712,"internally_reserved":0,"allocated":256765952,"data_stored":256189568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":256561449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,11,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":99762176,"num_objects":12178,"num_object_clones":0,"num_object_copies":48712,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":12124,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":12178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":97424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003340"},"pg_stats":[{"pgid":"1.f","version":"28'1302","reported_seq":1330,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.008502+0000","last_change":"2024-09-10T20:19:41.780800+0000","last_active":"2024-09-10T20:20:00.008502+0000","last_peered":"2024-09-10T20:20:00.008502+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:20:00.008502+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1302,"log_dups_size":0,"ondisk_log_size":1302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:00:26.995081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10665984,"num_objects":1302,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1302,"num_write_kb":10416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1308","reported_seq":1359,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.996470+0000","last_change":"2024-09-10T20:19:52.756178+0000","last_active":"2024-09-10T20:19:59.996470+0000","last_peered":"2024-09-10T20:19:59.996470+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:19:59.996470+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679456,"num_objects":1304,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1306,"num_write_kb":10444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1304}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1392","reported_seq":1454,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.866359+0000","last_change":"2024-09-10T20:19:53.348531+0000","last_active":"2024-09-10T20:20:01.866359+0000","last_peered":"2024-09-10T20:20:01.866359+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:20:01.866359+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1392,"log_dups_size":0,"ondisk_log_size":1392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.434390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11377232,"num_objects":1389,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1391,"num_write_kb":11118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1307","reported_seq":1357,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.004096+0000","last_change":"2024-09-10T20:19:53.542132+0000","last_active":"2024-09-10T20:20:00.004096+0000","last_peered":"2024-09-10T20:20:00.004096+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:20:00.004096+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671264,"num_objects":1303,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":10429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1303}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1277","reported_seq":1316,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.998785+0000","last_change":"2024-09-10T20:19:54.054504+0000","last_active":"2024-09-10T20:19:59.998785+0000","last_peered":"2024-09-10T20:19:59.998785+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:19:59.998785+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.832554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10433696,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":10197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1275","reported_seq":1304,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.960114+0000","last_change":"2024-09-10T20:19:41.823169+0000","last_active":"2024-09-10T20:19:59.960114+0000","last_peered":"2024-09-10T20:19:59.960114+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:19:59.960114+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1275,"log_dups_size":0,"ondisk_log_size":1275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:01.206261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10444800,"num_objects":1275,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":10200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'1284","reported_seq":1335,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.937047+0000","last_change":"2024-09-10T20:19:54.574637+0000","last_active":"2024-09-10T20:19:59.937047+0000","last_peered":"2024-09-10T20:19:59.937047+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:19:59.937047+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:55.166279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10466464,"num_objects":1278,"num_object_clones":0,"num_object_copies":5112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1282,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1355","reported_seq":1405,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.013727+0000","last_change":"2024-09-10T20:19:55.088547+0000","last_active":"2024-09-10T20:20:00.013727+0000","last_peered":"2024-09-10T20:20:00.013727+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:20:00.013727+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056288,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1353,"num_write_kb":10812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1350}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1288","reported_seq":1330,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.005278+0000","last_change":"2024-09-10T20:19:55.620606+0000","last_active":"2024-09-10T20:20:00.005278+0000","last_peered":"2024-09-10T20:20:00.005278+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:20:00.005278+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:55.851234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10479936,"num_objects":1280,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1285,"num_write_kb":10257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1334","reported_seq":1398,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.847644+0000","last_change":"2024-09-10T20:19:56.094118+0000","last_active":"2024-09-10T20:20:01.847644+0000","last_peered":"2024-09-10T20:20:01.847644+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:20:01.847644+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1334,"log_dups_size":0,"ondisk_log_size":1334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.498363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10908832,"num_objects":1332,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1332,"num_write_kb":10654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1332}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1274","reported_seq":1316,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.957844+0000","last_change":"2024-09-10T20:19:56.662872+0000","last_active":"2024-09-10T20:19:59.957844+0000","last_peered":"2024-09-10T20:19:59.957844+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:19:59.957844+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.901247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10391280,"num_objects":1269,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1271,"num_write_kb":10156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1301","reported_seq":1332,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.938523+0000","last_change":"2024-09-10T20:19:57.121670+0000","last_active":"2024-09-10T20:19:59.938523+0000","last_peered":"2024-09-10T20:19:59.938523+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:19:59.938523+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10648144,"num_objects":1300,"num_object_clones":0,"num_object_copies":5200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1300,"num_write_kb":10399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1363","reported_seq":1425,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.869740+0000","last_change":"2024-09-10T20:19:57.613197+0000","last_active":"2024-09-10T20:20:01.869740+0000","last_peered":"2024-09-10T20:20:01.869740+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:20:01.869740+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1363,"log_dups_size":0,"ondisk_log_size":1363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.554790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11156048,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":10895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1334","reported_seq":1373,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.958829+0000","last_change":"2024-09-10T20:19:58.151835+0000","last_active":"2024-09-10T20:19:59.958829+0000","last_peered":"2024-09-10T20:19:59.958829+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:19:59.958829+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1334,"log_dups_size":0,"ondisk_log_size":1334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10892448,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1332,"num_write_kb":10652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1370","reported_seq":1436,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.867626+0000","last_change":"2024-09-10T20:19:58.680200+0000","last_active":"2024-09-10T20:20:01.867626+0000","last_peered":"2024-09-10T20:20:01.867626+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:20:01.867626+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:59.507307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195552,"num_objects":1367,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1368,"num_write_kb":10941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1367}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1313","reported_seq":1363,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.002246+0000","last_change":"2024-09-10T20:19:59.268306+0000","last_active":"2024-09-10T20:20:00.002246+0000","last_peered":"2024-09-10T20:20:00.002246+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:20:00.002246+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10712224,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1311,"num_write_kb":10476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":172179648,"num_objects":21023,"num_object_clones":0,"num_object_copies":84092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21050,"num_write_kb":168282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259817472,"data_stored":259817472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21077,"ondisk_log_size":21077,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158688,"kb_used_data":82576,"kb_used_omap":4,"kb_used_meta":76091,"kb_avail":94213152,"statfs":{"total":96636764160,"available":96474267648,"internally_reserved":0,"allocated":84557824,"data_stored":84399268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":77917904},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163048,"kb_used_data":87448,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94208792,"statfs":{"total":96636764160,"available":96469803008,"internally_reserved":0,"allocated":89546752,"data_stored":89388196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":83604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152432,"kb_used_data":80544,"kb_used_omap":4,"kb_used_meta":71867,"kb_avail":94219408,"statfs":{"total":96636764160,"available":96480673792,"internally_reserved":0,"allocated":82477056,"data_stored":82318500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73592527},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84709376,"data_stored":84709376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90423296,"data_stored":90423296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84643840,"data_stored":84643840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:04.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:04.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 23 20 24138 24118 8.19144 5.5 0.020783 0.0190608 2024-09-10T20:20:04.902 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:04.902 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:04.922 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-10T20:20:03.884961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":172179648,"num_objects":21023,"num_object_clones":0,"num_object_copies":84092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21050,"num_write_kb":168282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21077,"ondisk_log_size":21077,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":501372,"kb_used_data":250748,"kb_used_omap":11,"kb_used_meta":250548,"kb_avail":376985988,"statfs":{"total":386547056640,"available":386033651712,"internally_reserved":0,"allocated":256765952,"data_stored":256189568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":256561449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,11,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":99762176,"num_objects":12178,"num_object_clones":0,"num_object_copies":48712,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":12124,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":12178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":97424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003340"},"pg_stats":[{"pgid":"1.f","version":"28'1302","reported_seq":1330,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.008502+0000","last_change":"2024-09-10T20:19:41.780800+0000","last_active":"2024-09-10T20:20:00.008502+0000","last_peered":"2024-09-10T20:20:00.008502+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:20:00.008502+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1302,"log_dups_size":0,"ondisk_log_size":1302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:00:26.995081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10665984,"num_objects":1302,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1302,"num_write_kb":10416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1308","reported_seq":1359,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.996470+0000","last_change":"2024-09-10T20:19:52.756178+0000","last_active":"2024-09-10T20:19:59.996470+0000","last_peered":"2024-09-10T20:19:59.996470+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:19:59.996470+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679456,"num_objects":1304,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1306,"num_write_kb":10444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1304}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1392","reported_seq":1454,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.866359+0000","last_change":"2024-09-10T20:19:53.348531+0000","last_active":"2024-09-10T20:20:01.866359+0000","last_peered":"2024-09-10T20:20:01.866359+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:20:01.866359+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1392,"log_dups_size":0,"ondisk_log_size":1392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.434390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11377232,"num_objects":1389,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1391,"num_write_kb":11118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1307","reported_seq":1357,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.004096+0000","last_change":"2024-09-10T20:19:53.542132+0000","last_active":"2024-09-10T20:20:00.004096+0000","last_peered":"2024-09-10T20:20:00.004096+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:20:00.004096+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671264,"num_objects":1303,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":10429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1303}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1277","reported_seq":1316,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.998785+0000","last_change":"2024-09-10T20:19:54.054504+0000","last_active":"2024-09-10T20:19:59.998785+0000","last_peered":"2024-09-10T20:19:59.998785+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:19:59.998785+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.832554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10433696,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":10197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1275","reported_seq":1304,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.960114+0000","last_change":"2024-09-10T20:19:41.823169+0000","last_active":"2024-09-10T20:19:59.960114+0000","last_peered":"2024-09-10T20:19:59.960114+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:19:59.960114+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1275,"log_dups_size":0,"ondisk_log_size":1275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:01.206261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10444800,"num_objects":1275,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":10200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'1284","reported_seq":1335,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.937047+0000","last_change":"2024-09-10T20:19:54.574637+0000","last_active":"2024-09-10T20:19:59.937047+0000","last_peered":"2024-09-10T20:19:59.937047+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:19:59.937047+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:55.166279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10466464,"num_objects":1278,"num_object_clones":0,"num_object_copies":5112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1282,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1355","reported_seq":1405,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.013727+0000","last_change":"2024-09-10T20:19:55.088547+0000","last_active":"2024-09-10T20:20:00.013727+0000","last_peered":"2024-09-10T20:20:00.013727+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:20:00.013727+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056288,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1353,"num_write_kb":10812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1350}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1288","reported_seq":1330,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.005278+0000","last_change":"2024-09-10T20:19:55.620606+0000","last_active":"2024-09-10T20:20:00.005278+0000","last_peered":"2024-09-10T20:20:00.005278+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:20:00.005278+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:55.851234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10479936,"num_objects":1280,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1285,"num_write_kb":10257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1334","reported_seq":1398,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.847644+0000","last_change":"2024-09-10T20:19:56.094118+0000","last_active":"2024-09-10T20:20:01.847644+0000","last_peered":"2024-09-10T20:20:01.847644+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:20:01.847644+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1334,"log_dups_size":0,"ondisk_log_size":1334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.498363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10908832,"num_objects":1332,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1332,"num_write_kb":10654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1332}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1274","reported_seq":1316,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.957844+0000","last_change":"2024-09-10T20:19:56.662872+0000","last_active":"2024-09-10T20:19:59.957844+0000","last_peered":"2024-09-10T20:19:59.957844+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:19:59.957844+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.901247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10391280,"num_objects":1269,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1271,"num_write_kb":10156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1301","reported_seq":1332,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.938523+0000","last_change":"2024-09-10T20:19:57.121670+0000","last_active":"2024-09-10T20:19:59.938523+0000","last_peered":"2024-09-10T20:19:59.938523+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:19:59.938523+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10648144,"num_objects":1300,"num_object_clones":0,"num_object_copies":5200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1300,"num_write_kb":10399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1363","reported_seq":1425,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.869740+0000","last_change":"2024-09-10T20:19:57.613197+0000","last_active":"2024-09-10T20:20:01.869740+0000","last_peered":"2024-09-10T20:20:01.869740+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:20:01.869740+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1363,"log_dups_size":0,"ondisk_log_size":1363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.554790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11156048,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":10895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1334","reported_seq":1373,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:19:59.958829+0000","last_change":"2024-09-10T20:19:58.151835+0000","last_active":"2024-09-10T20:19:59.958829+0000","last_peered":"2024-09-10T20:19:59.958829+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:19:59.958829+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1334,"log_dups_size":0,"ondisk_log_size":1334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10892448,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1332,"num_write_kb":10652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1370","reported_seq":1436,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:01.867626+0000","last_change":"2024-09-10T20:19:58.680200+0000","last_active":"2024-09-10T20:20:01.867626+0000","last_peered":"2024-09-10T20:20:01.867626+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:20:01.867626+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:59.507307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195552,"num_objects":1367,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1368,"num_write_kb":10941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1367}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1313","reported_seq":1363,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:00.002246+0000","last_change":"2024-09-10T20:19:59.268306+0000","last_active":"2024-09-10T20:20:00.002246+0000","last_peered":"2024-09-10T20:20:00.002246+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:20:00.002246+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10712224,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1311,"num_write_kb":10476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":172179648,"num_objects":21023,"num_object_clones":0,"num_object_copies":84092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21050,"num_write_kb":168282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259817472,"data_stored":259817472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21077,"ondisk_log_size":21077,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158688,"kb_used_data":82576,"kb_used_omap":4,"kb_used_meta":76091,"kb_avail":94213152,"statfs":{"total":96636764160,"available":96474267648,"internally_reserved":0,"allocated":84557824,"data_stored":84399268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":77917904},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163048,"kb_used_data":87448,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94208792,"statfs":{"total":96636764160,"available":96469803008,"internally_reserved":0,"allocated":89546752,"data_stored":89388196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":83604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152432,"kb_used_data":80544,"kb_used_omap":4,"kb_used_meta":71867,"kb_avail":94219408,"statfs":{"total":96636764160,"available":96480673792,"internally_reserved":0,"allocated":82477056,"data_stored":82318500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73592527},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84709376,"data_stored":84709376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90423296,"data_stored":90423296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84643840,"data_stored":84643840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:04.923 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-10T20:20:04.923 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-09-10T20:20:04.923 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-09-10T20:20:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-09-10T20:20:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.027 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-09-10T20:20:05.027 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-09-10T20:20:05.137 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-09-10T20:20:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.145 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-09-10T20:20:05.145 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-09-10T20:20:05.231 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-09-10T20:20:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.236 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-09-10T20:20:05.237 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-09-10T20:20:05.341 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-09-10T20:20:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.348 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-09-10T20:20:05.348 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-09-10T20:20:05.452 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-09-10T20:20:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.458 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-09-10T20:20:05.458 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-09-10T20:20:05.559 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-09-10T20:20:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.567 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-09-10T20:20:05.567 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-09-10T20:20:05.666 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-09-10T20:20:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.672 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-09-10T20:20:05.672 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-09-10T20:20:05.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 24 20 24962 24942 8.11834 6.4375 0.0257176 0.0192308 2024-09-10T20:20:05.787 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-09-10T20:20:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.797 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-09-10T20:20:05.798 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-09-10T20:20:05.886 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-09-10T20:20:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:05.895 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-09-10T20:20:05.895 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-09-10T20:20:05.992 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-09-10T20:20:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.002 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-09-10T20:20:06.003 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-09-10T20:20:06.099 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-09-10T20:20:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.103 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-09-10T20:20:06.103 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-09-10T20:20:06.202 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-09-10T20:20:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.209 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-09-10T20:20:06.209 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-09-10T20:20:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-09-10T20:20:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.318 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-09-10T20:20:06.318 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-09-10T20:20:06.418 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-09-10T20:20:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.433 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-09-10T20:20:06.433 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-09-10T20:20:06.539 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-09-10T20:20:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.547 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-09-10T20:20:06.547 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-09-10T20:20:06.620 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-09-10T20:20:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.622 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-09-10T20:20:06.622 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-09-10T20:20:06.667 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-09-10T20:20:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.669 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-09-10T20:20:06.670 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-09-10T20:20:06.714 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-09-10T20:20:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.716 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-09-10T20:20:06.717 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-09-10T20:20:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-09-10T20:20:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.765 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-09-10T20:20:06.766 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-09-10T20:20:06.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 25 20 25683 25663 8.01891 5.63281 0.00540714 0.019334 2024-09-10T20:20:06.853 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-09-10T20:20:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:06.865 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-09-10T20:20:06.866 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-09-10T20:20:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-09-10T20:20:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:07.016 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-09-10T20:20:07.017 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-09-10T20:20:07.107 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-09-10T20:20:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:07.116 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-09-10T20:20:07.116 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-09-10T20:20:07.207 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-09-10T20:20:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:07.212 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-09-10T20:20:07.212 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-09-10T20:20:07.313 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-09-10T20:20:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:07.324 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-09-10T20:20:07.324 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-09-10T20:20:07.427 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-09-10T20:20:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:07.435 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-09-10T20:20:07.435 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-09-10T20:20:07.531 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-09-10T20:20:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:07.538 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-09-10T20:20:07.538 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-09-10T20:20:07.589 INFO:teuthology.orchestra.run.smithi103.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-09-10T20:20:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:20:07.591 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-10T20:20:07.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-10T20:20:07.593 INFO:tasks.ceph.osd.1:Started 2024-09-10T20:20:07.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:config diff 2024-09-10T20:20:07.691 INFO:teuthology.orchestra.run.smithi103.stderr:config diff get 2024-09-10T20:20:07.692 INFO:teuthology.orchestra.run.smithi103.stderr:config get 2024-09-10T20:20:07.692 INFO:teuthology.orchestra.run.smithi103.stderr:config help [] 2024-09-10T20:20:07.692 INFO:teuthology.orchestra.run.smithi103.stderr:config set ... 2024-09-10T20:20:07.692 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-09-10T20:20:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:20:07.693 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-10T20:20:07.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 26 20 26461 26441 7.94424 6.07812 0.013739 0.0196574 2024-09-10T20:20:08.761 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:20:08.755+0000 7fc5a981c640 -1 Falling back to public interface 2024-09-10T20:20:08.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 27 20 27306 27286 7.8945 6.60156 0.018372 0.0197815 2024-09-10T20:20:09.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 28 20 28148 28128 7.84747 6.57812 0.0117751 0.0198999 2024-09-10T20:20:10.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 29 20 29162 29142 7.85001 7.92188 0.0174454 0.019896 2024-09-10T20:20:11.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 30 20 29962 29942 7.79666 6.25 0.0102191 0.0200321 2024-09-10T20:20:12.373 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:20:12.370+0000 7fc5a981c640 -1 osd.1 19 log_to_monitors true 2024-09-10T20:20:12.439 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:20:12.435+0000 7fc5a0fc8640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:20:12.695 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-10T20:20:12.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 31 20 30903 30883 7.78229 7.35156 0.0279648 0.0200638 2024-09-10T20:20:12.865 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:20:12.865 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-10T20:20:12.865 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-10T20:20:12.865 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:20:12.882 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-09-10T20:20:12.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:20:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:13.444 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":30,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:20:13.417830+0000","last_up_change":"2024-09-10T20:20:13.417830+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-10T20:18:53.939346+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":27,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":3732379198}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":3732379198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":3732379198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":3732379198}]},"public_addr":"172.21.15.103:6800/3732379198","cluster_addr":"172.21.15.103:6801/3732379198","heartbeat_back_addr":"172.21.15.103:6803/3732379198","heartbeat_front_addr":"172.21.15.103:6802/3732379198","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":30,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":2304361244}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":2304361244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":2304361244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":2304361244}]},"public_addr":"172.21.15.103:6812/2304361244","cluster_addr":"172.21.15.103:6813/2304361244","heartbeat_back_addr":"172.21.15.103:6815/2304361244","heartbeat_front_addr":"172.21.15.103:6814/2304361244","state":["exists","up"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":27,"up_thru":27,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":3618344502}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":3618344502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":3618344502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":3618344502}]},"public_addr":"172.21.15.103:6808/3618344502","cluster_addr":"172.21.15.103:6809/3618344502","heartbeat_back_addr":"172.21.15.103:6811/3618344502","heartbeat_front_addr":"172.21.15.103:6810/3618344502","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:20:13.465 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-09-10T20:20:13.465 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-10T20:20:13.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:13.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 32 20 31464 31444 7.67603 4.38281 0.0415972 0.0201303 2024-09-10T20:20:14.570 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:14.584 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-10T20:20:14.437811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":726568,"kb_used_data":371296,"kb_used_omap":14,"kb_used_meta":355185,"kb_avail":376760792,"statfs":{"total":386547056640,"available":385803051008,"internally_reserved":0,"allocated":380207104,"data_stored":379602766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":363710006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44589056,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":43544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.551931"},"pg_stats":[{"pgid":"1.f","version":"29'1966","reported_seq":1998,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.438700+0000","last_change":"2024-09-10T20:20:13.450100+0000","last_active":"2024-09-10T20:20:13.450100+0000","last_peered":"2024-09-10T20:20:13.438700+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:20:13.438700+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1966,"log_dups_size":0,"ondisk_log_size":1966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16105472,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":15728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1734","reported_seq":1785,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.010567+0000","last_change":"2024-09-10T20:19:52.756178+0000","last_active":"2024-09-10T20:20:09.010567+0000","last_peered":"2024-09-10T20:20:09.010567+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:20:09.010567+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1734,"log_dups_size":0,"ondisk_log_size":1734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14169248,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1732,"num_write_kb":13852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1730}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1886","reported_seq":1948,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.913250+0000","last_change":"2024-09-10T20:19:53.348531+0000","last_active":"2024-09-10T20:20:11.913250+0000","last_peered":"2024-09-10T20:20:11.913250+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:20:11.913250+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1886,"log_dups_size":0,"ondisk_log_size":1886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.434390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15424080,"num_objects":1883,"num_object_clones":0,"num_object_copies":7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1720","reported_seq":1771,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.039326+0000","last_change":"2024-09-10T20:19:53.542132+0000","last_active":"2024-09-10T20:20:09.039326+0000","last_peered":"2024-09-10T20:20:09.039326+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:20:09.039326+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":13733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'1985","reported_seq":2026,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.447183+0000","last_change":"2024-09-10T20:20:13.450855+0000","last_active":"2024-09-10T20:20:13.450855+0000","last_peered":"2024-09-10T20:20:13.447183+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:20:13.447183+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1985,"log_dups_size":0,"ondisk_log_size":1985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233632,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":15861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'1938","reported_seq":1970,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.437352+0000","last_change":"2024-09-10T20:20:13.451390+0000","last_active":"2024-09-10T20:20:13.451390+0000","last_peered":"2024-09-10T20:20:13.437352+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:20:13.437352+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1938,"log_dups_size":0,"ondisk_log_size":1938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1938,"num_write_kb":15504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'1961","reported_seq":2015,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.416166+0000","last_change":"2024-09-10T20:20:13.447557+0000","last_active":"2024-09-10T20:20:13.447557+0000","last_peered":"2024-09-10T20:20:13.416166+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:20:13.416166+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":15652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1779","reported_seq":1829,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.041843+0000","last_change":"2024-09-10T20:19:55.088547+0000","last_active":"2024-09-10T20:20:09.041843+0000","last_peered":"2024-09-10T20:20:09.041843+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:20:09.041843+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1774}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'1932","reported_seq":1977,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.446851+0000","last_change":"2024-09-10T20:20:13.454122+0000","last_active":"2024-09-10T20:20:13.454122+0000","last_peered":"2024-09-10T20:20:13.446851+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:20:13.446851+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1932,"log_dups_size":0,"ondisk_log_size":1932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15755584,"num_objects":1924,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1929,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1823","reported_seq":1887,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.919201+0000","last_change":"2024-09-10T20:19:56.094118+0000","last_active":"2024-09-10T20:20:11.919201+0000","last_peered":"2024-09-10T20:20:11.919201+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:20:11.919201+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1823,"log_dups_size":0,"ondisk_log_size":1823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.498363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14914720,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1821,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1821}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'1964","reported_seq":2008,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.442423+0000","last_change":"2024-09-10T20:20:13.454675+0000","last_active":"2024-09-10T20:20:13.454675+0000","last_peered":"2024-09-10T20:20:13.442423+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:20:13.442423+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16043760,"num_objects":1959,"num_object_clones":0,"num_object_copies":7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":15676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1729","reported_seq":1759,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.128660+0000","last_change":"2024-09-10T20:19:57.121670+0000","last_active":"2024-09-10T20:20:09.128660+0000","last_peered":"2024-09-10T20:20:09.128660+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:20:09.128660+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14154320,"num_objects":1728,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1869","reported_seq":1931,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.920654+0000","last_change":"2024-09-10T20:19:57.613197+0000","last_active":"2024-09-10T20:20:11.920654+0000","last_peered":"2024-09-10T20:20:11.920654+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:20:11.920654+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1869,"log_dups_size":0,"ondisk_log_size":1869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.554790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15301200,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1763","reported_seq":1802,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.124320+0000","last_change":"2024-09-10T20:19:58.151835+0000","last_active":"2024-09-10T20:20:09.124320+0000","last_peered":"2024-09-10T20:20:09.124320+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:20:09.124320+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406816,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":14084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1887","reported_seq":1953,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.899638+0000","last_change":"2024-09-10T20:19:58.680200+0000","last_active":"2024-09-10T20:20:11.899638+0000","last_peered":"2024-09-10T20:20:11.899638+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:20:11.899638+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1887,"log_dups_size":0,"ondisk_log_size":1887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:59.507307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1884}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1746","reported_seq":1796,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.005805+0000","last_change":"2024-09-10T20:19:59.268306+0000","last_active":"2024-09-10T20:20:09.005805+0000","last_peered":"2024-09-10T20:20:09.005805+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:20:09.005805+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1746,"log_dups_size":0,"ondisk_log_size":1746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:21:00.218351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259360,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1744,"num_write_kb":13940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381235200,"data_stored":381235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238296,"kb_used_data":126080,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94133544,"statfs":{"total":96636764160,"available":96392749056,"internally_reserved":0,"allocated":129105920,"data_stored":128938413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227376,"kb_used_data":119840,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94144464,"statfs":{"total":96636764160,"available":96403931136,"internally_reserved":0,"allocated":122716160,"data_stored":122557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":278528,"data_stored":163181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233088,"kb_used_data":125104,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":94138752,"statfs":{"total":96636764160,"available":96398082048,"internally_reserved":0,"allocated":128106496,"data_stored":127943568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":110554831},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123416576,"data_stored":123416576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:14.585 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:14.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 33 20 31464 31444 7.44342 0 - 0.0201303 2024-09-10T20:20:14.872 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:14.873 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:14.884 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-10T20:20:14.437811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":726568,"kb_used_data":371296,"kb_used_omap":14,"kb_used_meta":355185,"kb_avail":376760792,"statfs":{"total":386547056640,"available":385803051008,"internally_reserved":0,"allocated":380207104,"data_stored":379602766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":363710006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44589056,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":43544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.551931"},"pg_stats":[{"pgid":"1.f","version":"29'1966","reported_seq":1998,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.438700+0000","last_change":"2024-09-10T20:20:13.450100+0000","last_active":"2024-09-10T20:20:13.450100+0000","last_peered":"2024-09-10T20:20:13.438700+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:20:13.438700+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1966,"log_dups_size":0,"ondisk_log_size":1966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16105472,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":15728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1734","reported_seq":1785,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.010567+0000","last_change":"2024-09-10T20:19:52.756178+0000","last_active":"2024-09-10T20:20:09.010567+0000","last_peered":"2024-09-10T20:20:09.010567+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:20:09.010567+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1734,"log_dups_size":0,"ondisk_log_size":1734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14169248,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1732,"num_write_kb":13852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1730}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1886","reported_seq":1948,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.913250+0000","last_change":"2024-09-10T20:19:53.348531+0000","last_active":"2024-09-10T20:20:11.913250+0000","last_peered":"2024-09-10T20:20:11.913250+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:20:11.913250+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1886,"log_dups_size":0,"ondisk_log_size":1886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.434390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15424080,"num_objects":1883,"num_object_clones":0,"num_object_copies":7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1720","reported_seq":1771,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.039326+0000","last_change":"2024-09-10T20:19:53.542132+0000","last_active":"2024-09-10T20:20:09.039326+0000","last_peered":"2024-09-10T20:20:09.039326+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:20:09.039326+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":13733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'1985","reported_seq":2026,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.447183+0000","last_change":"2024-09-10T20:20:13.450855+0000","last_active":"2024-09-10T20:20:13.450855+0000","last_peered":"2024-09-10T20:20:13.447183+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:20:13.447183+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1985,"log_dups_size":0,"ondisk_log_size":1985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233632,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":15861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'1938","reported_seq":1970,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.437352+0000","last_change":"2024-09-10T20:20:13.451390+0000","last_active":"2024-09-10T20:20:13.451390+0000","last_peered":"2024-09-10T20:20:13.437352+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:20:13.437352+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1938,"log_dups_size":0,"ondisk_log_size":1938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1938,"num_write_kb":15504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'1961","reported_seq":2015,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.416166+0000","last_change":"2024-09-10T20:20:13.447557+0000","last_active":"2024-09-10T20:20:13.447557+0000","last_peered":"2024-09-10T20:20:13.416166+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:20:13.416166+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":15652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1779","reported_seq":1829,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.041843+0000","last_change":"2024-09-10T20:19:55.088547+0000","last_active":"2024-09-10T20:20:09.041843+0000","last_peered":"2024-09-10T20:20:09.041843+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:20:09.041843+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1774}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'1932","reported_seq":1977,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.446851+0000","last_change":"2024-09-10T20:20:13.454122+0000","last_active":"2024-09-10T20:20:13.454122+0000","last_peered":"2024-09-10T20:20:13.446851+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:20:13.446851+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1932,"log_dups_size":0,"ondisk_log_size":1932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15755584,"num_objects":1924,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1929,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1823","reported_seq":1887,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.919201+0000","last_change":"2024-09-10T20:19:56.094118+0000","last_active":"2024-09-10T20:20:11.919201+0000","last_peered":"2024-09-10T20:20:11.919201+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:20:11.919201+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1823,"log_dups_size":0,"ondisk_log_size":1823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.498363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14914720,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1821,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1821}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'1964","reported_seq":2008,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.442423+0000","last_change":"2024-09-10T20:20:13.454675+0000","last_active":"2024-09-10T20:20:13.454675+0000","last_peered":"2024-09-10T20:20:13.442423+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:20:13.442423+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16043760,"num_objects":1959,"num_object_clones":0,"num_object_copies":7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":15676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1729","reported_seq":1759,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.128660+0000","last_change":"2024-09-10T20:19:57.121670+0000","last_active":"2024-09-10T20:20:09.128660+0000","last_peered":"2024-09-10T20:20:09.128660+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:20:09.128660+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14154320,"num_objects":1728,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1869","reported_seq":1931,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.920654+0000","last_change":"2024-09-10T20:19:57.613197+0000","last_active":"2024-09-10T20:20:11.920654+0000","last_peered":"2024-09-10T20:20:11.920654+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:20:11.920654+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1869,"log_dups_size":0,"ondisk_log_size":1869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.554790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15301200,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1763","reported_seq":1802,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.124320+0000","last_change":"2024-09-10T20:19:58.151835+0000","last_active":"2024-09-10T20:20:09.124320+0000","last_peered":"2024-09-10T20:20:09.124320+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:20:09.124320+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406816,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":14084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1887","reported_seq":1953,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.899638+0000","last_change":"2024-09-10T20:19:58.680200+0000","last_active":"2024-09-10T20:20:11.899638+0000","last_peered":"2024-09-10T20:20:11.899638+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:20:11.899638+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1887,"log_dups_size":0,"ondisk_log_size":1887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:59.507307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1884}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1746","reported_seq":1796,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.005805+0000","last_change":"2024-09-10T20:19:59.268306+0000","last_active":"2024-09-10T20:20:09.005805+0000","last_peered":"2024-09-10T20:20:09.005805+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:20:09.005805+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1746,"log_dups_size":0,"ondisk_log_size":1746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:21:00.218351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259360,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1744,"num_write_kb":13940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381235200,"data_stored":381235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238296,"kb_used_data":126080,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94133544,"statfs":{"total":96636764160,"available":96392749056,"internally_reserved":0,"allocated":129105920,"data_stored":128938413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227376,"kb_used_data":119840,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94144464,"statfs":{"total":96636764160,"available":96403931136,"internally_reserved":0,"allocated":122716160,"data_stored":122557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":278528,"data_stored":163181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233088,"kb_used_data":125104,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":94138752,"statfs":{"total":96636764160,"available":96398082048,"internally_reserved":0,"allocated":128106496,"data_stored":127943568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":110554831},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123416576,"data_stored":123416576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:14.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:15.166 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:15.166 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:15.178 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-10T20:20:14.437811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":726568,"kb_used_data":371296,"kb_used_omap":14,"kb_used_meta":355185,"kb_avail":376760792,"statfs":{"total":386547056640,"available":385803051008,"internally_reserved":0,"allocated":380207104,"data_stored":379602766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":363710006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44589056,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":43544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.551931"},"pg_stats":[{"pgid":"1.f","version":"29'1966","reported_seq":1998,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.438700+0000","last_change":"2024-09-10T20:20:13.450100+0000","last_active":"2024-09-10T20:20:13.450100+0000","last_peered":"2024-09-10T20:20:13.438700+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:20:13.438700+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1966,"log_dups_size":0,"ondisk_log_size":1966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16105472,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":15728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1734","reported_seq":1785,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.010567+0000","last_change":"2024-09-10T20:19:52.756178+0000","last_active":"2024-09-10T20:20:09.010567+0000","last_peered":"2024-09-10T20:20:09.010567+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:20:09.010567+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1734,"log_dups_size":0,"ondisk_log_size":1734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14169248,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1732,"num_write_kb":13852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1730}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1886","reported_seq":1948,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.913250+0000","last_change":"2024-09-10T20:19:53.348531+0000","last_active":"2024-09-10T20:20:11.913250+0000","last_peered":"2024-09-10T20:20:11.913250+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:20:11.913250+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1886,"log_dups_size":0,"ondisk_log_size":1886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.434390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15424080,"num_objects":1883,"num_object_clones":0,"num_object_copies":7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1720","reported_seq":1771,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.039326+0000","last_change":"2024-09-10T20:19:53.542132+0000","last_active":"2024-09-10T20:20:09.039326+0000","last_peered":"2024-09-10T20:20:09.039326+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:20:09.039326+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":13733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'1985","reported_seq":2026,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.447183+0000","last_change":"2024-09-10T20:20:13.450855+0000","last_active":"2024-09-10T20:20:13.450855+0000","last_peered":"2024-09-10T20:20:13.447183+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:20:13.447183+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1985,"log_dups_size":0,"ondisk_log_size":1985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233632,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":15861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'1938","reported_seq":1970,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.437352+0000","last_change":"2024-09-10T20:20:13.451390+0000","last_active":"2024-09-10T20:20:13.451390+0000","last_peered":"2024-09-10T20:20:13.437352+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:20:13.437352+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1938,"log_dups_size":0,"ondisk_log_size":1938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1938,"num_write_kb":15504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'1961","reported_seq":2015,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.416166+0000","last_change":"2024-09-10T20:20:13.447557+0000","last_active":"2024-09-10T20:20:13.447557+0000","last_peered":"2024-09-10T20:20:13.416166+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:20:13.416166+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":15652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1779","reported_seq":1829,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.041843+0000","last_change":"2024-09-10T20:19:55.088547+0000","last_active":"2024-09-10T20:20:09.041843+0000","last_peered":"2024-09-10T20:20:09.041843+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:20:09.041843+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1774}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'1932","reported_seq":1977,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.446851+0000","last_change":"2024-09-10T20:20:13.454122+0000","last_active":"2024-09-10T20:20:13.454122+0000","last_peered":"2024-09-10T20:20:13.446851+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:20:13.446851+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1932,"log_dups_size":0,"ondisk_log_size":1932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15755584,"num_objects":1924,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1929,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1823","reported_seq":1887,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.919201+0000","last_change":"2024-09-10T20:19:56.094118+0000","last_active":"2024-09-10T20:20:11.919201+0000","last_peered":"2024-09-10T20:20:11.919201+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:20:11.919201+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1823,"log_dups_size":0,"ondisk_log_size":1823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.498363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14914720,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1821,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1821}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'1964","reported_seq":2008,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.442423+0000","last_change":"2024-09-10T20:20:13.454675+0000","last_active":"2024-09-10T20:20:13.454675+0000","last_peered":"2024-09-10T20:20:13.442423+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:20:13.442423+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16043760,"num_objects":1959,"num_object_clones":0,"num_object_copies":7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":15676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1729","reported_seq":1759,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.128660+0000","last_change":"2024-09-10T20:19:57.121670+0000","last_active":"2024-09-10T20:20:09.128660+0000","last_peered":"2024-09-10T20:20:09.128660+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:20:09.128660+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14154320,"num_objects":1728,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1869","reported_seq":1931,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.920654+0000","last_change":"2024-09-10T20:19:57.613197+0000","last_active":"2024-09-10T20:20:11.920654+0000","last_peered":"2024-09-10T20:20:11.920654+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:20:11.920654+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1869,"log_dups_size":0,"ondisk_log_size":1869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.554790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15301200,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1763","reported_seq":1802,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.124320+0000","last_change":"2024-09-10T20:19:58.151835+0000","last_active":"2024-09-10T20:20:09.124320+0000","last_peered":"2024-09-10T20:20:09.124320+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:20:09.124320+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406816,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":14084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1887","reported_seq":1953,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.899638+0000","last_change":"2024-09-10T20:19:58.680200+0000","last_active":"2024-09-10T20:20:11.899638+0000","last_peered":"2024-09-10T20:20:11.899638+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:20:11.899638+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1887,"log_dups_size":0,"ondisk_log_size":1887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:59.507307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1884}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1746","reported_seq":1796,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.005805+0000","last_change":"2024-09-10T20:19:59.268306+0000","last_active":"2024-09-10T20:20:09.005805+0000","last_peered":"2024-09-10T20:20:09.005805+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:20:09.005805+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1746,"log_dups_size":0,"ondisk_log_size":1746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:21:00.218351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259360,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1744,"num_write_kb":13940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381235200,"data_stored":381235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238296,"kb_used_data":126080,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94133544,"statfs":{"total":96636764160,"available":96392749056,"internally_reserved":0,"allocated":129105920,"data_stored":128938413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227376,"kb_used_data":119840,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94144464,"statfs":{"total":96636764160,"available":96403931136,"internally_reserved":0,"allocated":122716160,"data_stored":122557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":278528,"data_stored":163181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233088,"kb_used_data":125104,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":94138752,"statfs":{"total":96636764160,"available":96398082048,"internally_reserved":0,"allocated":128106496,"data_stored":127943568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":110554831},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123416576,"data_stored":123416576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "29'1966", 'reported_seq': 1998, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-10T20:20:13.438700+0000', 'last_change': '2024-09-10T20:20:13.450100+0000', 'last_active': '2024-09-10T20:20:13.450100+0000', 'last_peered': '2024-09-10T20:20:13.438700+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:19:31.152677+0000', 'last_became_peered': '2024-09-10T20:19:31.152677+0000', 'last_unstale': '2024-09-10T20:20:13.438700+0000', 'last_undegraded': '2024-09-10T20:19:41.780614+0000', 'last_fullsized': '2024-09-10T20:19:31.146438+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1966, 'log_dups_size': 0, 'ondisk_log_size': 1966, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16105472, 'num_objects': 1966, 'num_object_clones': 0, 'num_object_copies': 7864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1966, 'num_write_kb': 15728, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "28'1734", 'reported_seq': 1785, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:09.010567+0000', 'last_change': '2024-09-10T20:19:52.756178+0000', 'last_active': '2024-09-10T20:20:09.010567+0000', 'last_peered': '2024-09-10T20:20:09.010567+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:19:31.155250+0000', 'last_became_peered': '2024-09-10T20:19:31.155250+0000', 'last_unstale': '2024-09-10T20:20:09.010567+0000', 'last_undegraded': '2024-09-10T20:19:31.146570+0000', 'last_fullsized': '2024-09-10T20:19:31.146215+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1734, 'log_dups_size': 0, 'ondisk_log_size': 1734, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14169248, 'num_objects': 1730, 'num_object_clones': 0, 'num_object_copies': 6920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1730, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1732, 'num_write_kb': 13852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1730}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "28'1886", 'reported_seq': 1948, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:11.913250+0000', 'last_change': '2024-09-10T20:19:53.348531+0000', 'last_active': '2024-09-10T20:20:11.913250+0000', 'last_peered': '2024-09-10T20:20:11.913250+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:19:31.153825+0000', 'last_became_peered': '2024-09-10T20:19:31.153825+0000', 'last_unstale': '2024-09-10T20:20:11.913250+0000', 'last_undegraded': '2024-09-10T20:19:31.144878+0000', 'last_fullsized': '2024-09-10T20:19:31.144138+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1886, 'log_dups_size': 0, 'ondisk_log_size': 1886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:53.434390+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15424080, 'num_objects': 1883, 'num_object_clones': 0, 'num_object_copies': 7532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1883, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1885, 'num_write_kb': 15070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 2147483647, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1883}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "28'1720", 'reported_seq': 1771, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:09.039326+0000', 'last_change': '2024-09-10T20:19:53.542132+0000', 'last_active': '2024-09-10T20:20:09.039326+0000', 'last_peered': '2024-09-10T20:20:09.039326+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:19:31.154604+0000', 'last_became_peered': '2024-09-10T20:19:31.154604+0000', 'last_unstale': '2024-09-10T20:20:09.039326+0000', 'last_undegraded': '2024-09-10T20:19:31.144137+0000', 'last_fullsized': '2024-09-10T20:19:31.142952+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1720, 'log_dups_size': 0, 'ondisk_log_size': 1720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14054560, 'num_objects': 1716, 'num_object_clones': 0, 'num_object_copies': 6864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1718, 'num_write_kb': 13733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1716}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:15.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "29'1985", 'reported_seq': 2026, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-10T20:20:13.447183+0000', 'last_change': '2024-09-10T20:20:13.450855+0000', 'last_active': '2024-09-10T20:20:13.450855+0000', 'last_peered': '2024-09-10T20:20:13.447183+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:19:31.154652+0000', 'last_became_peered': '2024-09-10T20:19:31.154652+0000', 'last_unstale': '2024-09-10T20:20:13.447183+0000', 'last_undegraded': '2024-09-10T20:19:31.147348+0000', 'last_fullsized': '2024-09-10T20:19:31.147057+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1985, 'log_dups_size': 0, 'ondisk_log_size': 1985, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16233632, 'num_objects': 1982, 'num_object_clones': 0, 'num_object_copies': 7928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1983, 'num_write_kb': 15861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 1, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "29'1938", 'reported_seq': 1970, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-10T20:20:13.437352+0000', 'last_change': '2024-09-10T20:20:13.451390+0000', 'last_active': '2024-09-10T20:20:13.451390+0000', 'last_peered': '2024-09-10T20:20:13.437352+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:19:31.156062+0000', 'last_became_peered': '2024-09-10T20:19:31.156062+0000', 'last_unstale': '2024-09-10T20:20:13.437352+0000', 'last_undegraded': '2024-09-10T20:19:41.791102+0000', 'last_fullsized': '2024-09-10T20:19:31.147166+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1938, 'log_dups_size': 0, 'ondisk_log_size': 1938, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15876096, 'num_objects': 1938, 'num_object_clones': 0, 'num_object_copies': 7752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1938, 'num_write_kb': 15504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "29'1961", 'reported_seq': 2015, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-10T20:20:13.416166+0000', 'last_change': '2024-09-10T20:20:13.447557+0000', 'last_active': '2024-09-10T20:20:13.447557+0000', 'last_peered': '2024-09-10T20:20:13.416166+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:19:31.155101+0000', 'last_became_peered': '2024-09-10T20:19:31.155101+0000', 'last_unstale': '2024-09-10T20:20:13.416166+0000', 'last_undegraded': '2024-09-10T20:19:31.146566+0000', 'last_fullsized': '2024-09-10T20:19:31.146192+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1961, 'log_dups_size': 0, 'ondisk_log_size': 1961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16012448, 'num_objects': 1955, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1959, 'num_write_kb': 15652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "28'1779", 'reported_seq': 1829, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:09.041843+0000', 'last_change': '2024-09-10T20:19:55.088547+0000', 'last_active': '2024-09-10T20:20:09.041843+0000', 'last_peered': '2024-09-10T20:20:09.041843+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:19:31.156208+0000', 'last_became_peered': '2024-09-10T20:19:31.156208+0000', 'last_unstale': '2024-09-10T20:20:09.041843+0000', 'last_undegraded': '2024-09-10T20:19:31.146143+0000', 'last_fullsized': '2024-09-10T20:19:31.145598+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1779, 'log_dups_size': 0, 'ondisk_log_size': 1779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14529696, 'num_objects': 1774, 'num_object_clones': 0, 'num_object_copies': 7096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1774, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1777, 'num_write_kb': 14204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1774}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "29'1932", 'reported_seq': 1977, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-10T20:20:13.446851+0000', 'last_change': '2024-09-10T20:20:13.454122+0000', 'last_active': '2024-09-10T20:20:13.454122+0000', 'last_peered': '2024-09-10T20:20:13.446851+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:19:31.154204+0000', 'last_became_peered': '2024-09-10T20:19:31.154204+0000', 'last_unstale': '2024-09-10T20:20:13.446851+0000', 'last_undegraded': '2024-09-10T20:19:31.147561+0000', 'last_fullsized': '2024-09-10T20:19:31.147205+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1932, 'log_dups_size': 0, 'ondisk_log_size': 1932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15755584, 'num_objects': 1924, 'num_object_clones': 0, 'num_object_copies': 7696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1929, 'num_write_kb': 15409, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "28'1823", 'reported_seq': 1887, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:11.919201+0000', 'last_change': '2024-09-10T20:19:56.094118+0000', 'last_active': '2024-09-10T20:20:11.919201+0000', 'last_peered': '2024-09-10T20:20:11.919201+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:19:31.151201+0000', 'last_became_peered': '2024-09-10T20:19:31.151201+0000', 'last_unstale': '2024-09-10T20:20:11.919201+0000', 'last_undegraded': '2024-09-10T20:19:31.144546+0000', 'last_fullsized': '2024-09-10T20:19:31.144183+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1823, 'log_dups_size': 0, 'ondisk_log_size': 1823, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:56.498363+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14914720, 'num_objects': 1821, 'num_object_clones': 0, 'num_object_copies': 7284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1821, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1821, 'num_write_kb': 14566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 2147483647], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1821}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:15.180 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "29'1964", 'reported_seq': 2008, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-10T20:20:13.442423+0000', 'last_change': '2024-09-10T20:20:13.454675+0000', 'last_active': '2024-09-10T20:20:13.454675+0000', 'last_peered': '2024-09-10T20:20:13.442423+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:19:31.153021+0000', 'last_became_peered': '2024-09-10T20:19:31.153021+0000', 'last_unstale': '2024-09-10T20:20:13.442423+0000', 'last_undegraded': '2024-09-10T20:19:31.145240+0000', 'last_fullsized': '2024-09-10T20:19:31.144893+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1964, 'log_dups_size': 0, 'ondisk_log_size': 1964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16043760, 'num_objects': 1959, 'num_object_clones': 0, 'num_object_copies': 7836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1959, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1961, 'num_write_kb': 15676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "28'1729", 'reported_seq': 1759, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:09.128660+0000', 'last_change': '2024-09-10T20:19:57.121670+0000', 'last_active': '2024-09-10T20:20:09.128660+0000', 'last_peered': '2024-09-10T20:20:09.128660+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:19:31.154982+0000', 'last_became_peered': '2024-09-10T20:19:31.154982+0000', 'last_unstale': '2024-09-10T20:20:09.128660+0000', 'last_undegraded': '2024-09-10T20:19:31.148209+0000', 'last_fullsized': '2024-09-10T20:19:31.147914+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1729, 'log_dups_size': 0, 'ondisk_log_size': 1729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14154320, 'num_objects': 1728, 'num_object_clones': 0, 'num_object_copies': 6912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1728, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1728, 'num_write_kb': 13823, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1728}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "28'1869", 'reported_seq': 1931, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:11.920654+0000', 'last_change': '2024-09-10T20:19:57.613197+0000', 'last_active': '2024-09-10T20:20:11.920654+0000', 'last_peered': '2024-09-10T20:20:11.920654+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:19:31.154174+0000', 'last_became_peered': '2024-09-10T20:19:31.154174+0000', 'last_unstale': '2024-09-10T20:20:11.920654+0000', 'last_undegraded': '2024-09-10T20:19:31.144874+0000', 'last_fullsized': '2024-09-10T20:19:31.143751+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1869, 'log_dups_size': 0, 'ondisk_log_size': 1869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:58.554790+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15301200, 'num_objects': 1868, 'num_object_clones': 0, 'num_object_copies': 7472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1868, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1868, 'num_write_kb': 14943, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1868}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "28'1763", 'reported_seq': 1802, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:09.124320+0000', 'last_change': '2024-09-10T20:19:58.151835+0000', 'last_active': '2024-09-10T20:20:09.124320+0000', 'last_peered': '2024-09-10T20:20:09.124320+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:19:31.155530+0000', 'last_became_peered': '2024-09-10T20:19:31.155530+0000', 'last_unstale': '2024-09-10T20:20:09.124320+0000', 'last_undegraded': '2024-09-10T20:19:31.148111+0000', 'last_fullsized': '2024-09-10T20:19:31.147870+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1763, 'log_dups_size': 0, 'ondisk_log_size': 1763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14406816, 'num_objects': 1759, 'num_object_clones': 0, 'num_object_copies': 7036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1759, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1761, 'num_write_kb': 14084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1759}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "28'1887", 'reported_seq': 1953, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:11.899638+0000', 'last_change': '2024-09-10T20:19:58.680200+0000', 'last_active': '2024-09-10T20:20:11.899638+0000', 'last_peered': '2024-09-10T20:20:11.899638+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:19:31.151244+0000', 'last_became_peered': '2024-09-10T20:19:31.151244+0000', 'last_unstale': '2024-09-10T20:20:11.899638+0000', 'last_undegraded': '2024-09-10T20:19:31.144134+0000', 'last_fullsized': '2024-09-10T20:19:31.143752+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1887, 'log_dups_size': 0, 'ondisk_log_size': 1887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:20:59.507307+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15430816, 'num_objects': 1884, 'num_object_clones': 0, 'num_object_copies': 7536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1885, 'num_write_kb': 15077, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1884}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:15.181 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "28'1746", 'reported_seq': 1796, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-10T20:20:09.005805+0000', 'last_change': '2024-09-10T20:19:59.268306+0000', 'last_active': '2024-09-10T20:20:09.005805+0000', 'last_peered': '2024-09-10T20:20:09.005805+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:19:31.155468+0000', 'last_became_peered': '2024-09-10T20:19:31.155468+0000', 'last_unstale': '2024-09-10T20:20:09.005805+0000', 'last_undegraded': '2024-09-10T20:19:31.145015+0000', 'last_fullsized': '2024-09-10T20:19:31.144654+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1746, 'log_dups_size': 0, 'ondisk_log_size': 1746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-10T20:21:00.218351+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14259360, 'num_objects': 1741, 'num_object_clones': 0, 'num_object_copies': 6964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1741, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1744, 'num_write_kb': 13940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1741}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:15.182 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:15.630 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:15.631 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28138/112552 objects degraded (25.000%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":104,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":16}],"num_pgs":16,"num_pools":1,"num_objects":28138,"data_bytes":230465728,"bytes_used":695582720,"bytes_avail":385851473920,"bytes_total":386547056640,"degraded_objects":28138,"degraded_total":112552,"degraded_ratio":0.25,"write_bytes_sec":4980649,"read_op_per_sec":0,"write_op_per_sec":607},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-10T20:19:43.879327+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:15.643 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:20:15.643 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:15.772 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 34 20 31464 31444 7.2245 0 - 0.0201303 2024-09-10T20:20:16.027 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:16.027 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:16.048 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-10T20:20:15.887506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":726568,"kb_used_data":371296,"kb_used_omap":14,"kb_used_meta":355185,"kb_avail":376760792,"statfs":{"total":386547056640,"available":385803051008,"internally_reserved":0,"allocated":380207104,"data_stored":379602766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":363710006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21168128,"num_objects":2584,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-9140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":20672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000748"},"pg_stats":[{"pgid":"1.f","version":"29'1966","reported_seq":1998,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.438700+0000","last_change":"2024-09-10T20:20:13.450100+0000","last_active":"2024-09-10T20:20:13.450100+0000","last_peered":"2024-09-10T20:20:13.438700+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:20:13.438700+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1966,"log_dups_size":0,"ondisk_log_size":1966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16105472,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":15728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1734","reported_seq":1785,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.010567+0000","last_change":"2024-09-10T20:19:52.756178+0000","last_active":"2024-09-10T20:20:09.010567+0000","last_peered":"2024-09-10T20:20:09.010567+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:20:09.010567+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1734,"log_dups_size":0,"ondisk_log_size":1734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14169248,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1732,"num_write_kb":13852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1730}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1886","reported_seq":1948,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.913250+0000","last_change":"2024-09-10T20:19:53.348531+0000","last_active":"2024-09-10T20:20:11.913250+0000","last_peered":"2024-09-10T20:20:11.913250+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:19:31.153825+0000","last_became_peered":"2024-09-10T20:19:31.153825+0000","last_unstale":"2024-09-10T20:20:11.913250+0000","last_undegraded":"2024-09-10T20:19:31.144878+0000","last_fullsized":"2024-09-10T20:19:31.144138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1886,"log_dups_size":0,"ondisk_log_size":1886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.434390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15424080,"num_objects":1883,"num_object_clones":0,"num_object_copies":7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1720","reported_seq":1771,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.039326+0000","last_change":"2024-09-10T20:19:53.542132+0000","last_active":"2024-09-10T20:20:09.039326+0000","last_peered":"2024-09-10T20:20:09.039326+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:20:09.039326+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":13733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'1985","reported_seq":2026,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.447183+0000","last_change":"2024-09-10T20:20:13.450855+0000","last_active":"2024-09-10T20:20:13.450855+0000","last_peered":"2024-09-10T20:20:13.447183+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:20:13.447183+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1985,"log_dups_size":0,"ondisk_log_size":1985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233632,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":15861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'1938","reported_seq":1970,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.437352+0000","last_change":"2024-09-10T20:20:13.451390+0000","last_active":"2024-09-10T20:20:13.451390+0000","last_peered":"2024-09-10T20:20:13.437352+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:20:13.437352+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1938,"log_dups_size":0,"ondisk_log_size":1938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1938,"num_write_kb":15504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'1961","reported_seq":2015,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.416166+0000","last_change":"2024-09-10T20:20:13.447557+0000","last_active":"2024-09-10T20:20:13.447557+0000","last_peered":"2024-09-10T20:20:13.416166+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:20:13.416166+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":15652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1779","reported_seq":1829,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.041843+0000","last_change":"2024-09-10T20:19:55.088547+0000","last_active":"2024-09-10T20:20:09.041843+0000","last_peered":"2024-09-10T20:20:09.041843+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:20:09.041843+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1774}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'1932","reported_seq":1977,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.446851+0000","last_change":"2024-09-10T20:20:13.454122+0000","last_active":"2024-09-10T20:20:13.454122+0000","last_peered":"2024-09-10T20:20:13.446851+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:20:13.446851+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1932,"log_dups_size":0,"ondisk_log_size":1932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15755584,"num_objects":1924,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1929,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1823","reported_seq":1887,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.919201+0000","last_change":"2024-09-10T20:19:56.094118+0000","last_active":"2024-09-10T20:20:11.919201+0000","last_peered":"2024-09-10T20:20:11.919201+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:19:31.151201+0000","last_became_peered":"2024-09-10T20:19:31.151201+0000","last_unstale":"2024-09-10T20:20:11.919201+0000","last_undegraded":"2024-09-10T20:19:31.144546+0000","last_fullsized":"2024-09-10T20:19:31.144183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1823,"log_dups_size":0,"ondisk_log_size":1823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.498363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14914720,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1821,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1821}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'1964","reported_seq":2008,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.442423+0000","last_change":"2024-09-10T20:20:13.454675+0000","last_active":"2024-09-10T20:20:13.454675+0000","last_peered":"2024-09-10T20:20:13.442423+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:20:13.442423+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16043760,"num_objects":1959,"num_object_clones":0,"num_object_copies":7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":15676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1729","reported_seq":1759,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.128660+0000","last_change":"2024-09-10T20:19:57.121670+0000","last_active":"2024-09-10T20:20:09.128660+0000","last_peered":"2024-09-10T20:20:09.128660+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:20:09.128660+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14154320,"num_objects":1728,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1869","reported_seq":1931,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.920654+0000","last_change":"2024-09-10T20:19:57.613197+0000","last_active":"2024-09-10T20:20:11.920654+0000","last_peered":"2024-09-10T20:20:11.920654+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:19:31.154174+0000","last_became_peered":"2024-09-10T20:19:31.154174+0000","last_unstale":"2024-09-10T20:20:11.920654+0000","last_undegraded":"2024-09-10T20:19:31.144874+0000","last_fullsized":"2024-09-10T20:19:31.143751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1869,"log_dups_size":0,"ondisk_log_size":1869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.554790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15301200,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1763","reported_seq":1802,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.124320+0000","last_change":"2024-09-10T20:19:58.151835+0000","last_active":"2024-09-10T20:20:09.124320+0000","last_peered":"2024-09-10T20:20:09.124320+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:20:09.124320+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406816,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":14084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1887","reported_seq":1953,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:11.899638+0000","last_change":"2024-09-10T20:19:58.680200+0000","last_active":"2024-09-10T20:20:11.899638+0000","last_peered":"2024-09-10T20:20:11.899638+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:19:31.151244+0000","last_became_peered":"2024-09-10T20:19:31.151244+0000","last_unstale":"2024-09-10T20:20:11.899638+0000","last_undegraded":"2024-09-10T20:19:31.144134+0000","last_fullsized":"2024-09-10T20:19:31.143752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1887,"log_dups_size":0,"ondisk_log_size":1887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:59.507307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":15077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1884}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1746","reported_seq":1796,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.005805+0000","last_change":"2024-09-10T20:19:59.268306+0000","last_active":"2024-09-10T20:20:09.005805+0000","last_peered":"2024-09-10T20:20:09.005805+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:20:09.005805+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1746,"log_dups_size":0,"ondisk_log_size":1746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:21:00.218351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259360,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1744,"num_write_kb":13940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242671808,"num_objects":29628,"num_object_clones":0,"num_object_copies":118512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29655,"num_write_kb":237122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381235200,"data_stored":381235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29682,"ondisk_log_size":29682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238296,"kb_used_data":126080,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94133544,"statfs":{"total":96636764160,"available":96392749056,"internally_reserved":0,"allocated":129105920,"data_stored":128938413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227376,"kb_used_data":119840,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94144464,"statfs":{"total":96636764160,"available":96403931136,"internally_reserved":0,"allocated":122716160,"data_stored":122557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":278528,"data_stored":163181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233088,"kb_used_data":125104,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":94138752,"statfs":{"total":96636764160,"available":96398082048,"internally_reserved":0,"allocated":128106496,"data_stored":127943568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":110554831},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123416576,"data_stored":123416576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:16.773 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 35 20 31777 31757 7.0879 0.815104 0.0207244 0.0220302 2024-09-10T20:20:17.773 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 36 20 32279 32259 6.99995 3.92188 0.04838 0.0223045 2024-09-10T20:20:18.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 37 20 32783 32763 6.91718 3.9375 0.0425331 0.0225746 2024-09-10T20:20:19.051 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:19.626 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:19.627 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:19.645 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-10T20:20:17.889516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":246276288,"num_objects":30068,"num_object_clones":0,"num_object_copies":120272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":240642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30122,"ondisk_log_size":30122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":758528,"kb_used_data":381040,"kb_used_omap":14,"kb_used_meta":377393,"kb_avail":376728832,"statfs":{"total":386547056640,"available":385770323968,"internally_reserved":0,"allocated":390184960,"data_stored":389502563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":386450998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15810560,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-9909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1930,"num_write_kb":15440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.468522"},"pg_stats":[{"pgid":"1.f","version":"29'1966","reported_seq":1998,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.438700+0000","last_change":"2024-09-10T20:20:13.450100+0000","last_active":"2024-09-10T20:20:13.450100+0000","last_peered":"2024-09-10T20:20:13.438700+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.152677+0000","last_became_peered":"2024-09-10T20:19:31.152677+0000","last_unstale":"2024-09-10T20:20:13.438700+0000","last_undegraded":"2024-09-10T20:19:41.780614+0000","last_fullsized":"2024-09-10T20:19:31.146438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1966,"log_dups_size":0,"ondisk_log_size":1966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16105472,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":15728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1734","reported_seq":1785,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.010567+0000","last_change":"2024-09-10T20:19:52.756178+0000","last_active":"2024-09-10T20:20:09.010567+0000","last_peered":"2024-09-10T20:20:09.010567+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:19:31.155250+0000","last_became_peered":"2024-09-10T20:19:31.155250+0000","last_unstale":"2024-09-10T20:20:09.010567+0000","last_undegraded":"2024-09-10T20:19:31.146570+0000","last_fullsized":"2024-09-10T20:19:31.146215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1734,"log_dups_size":0,"ondisk_log_size":1734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:53.153115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14169248,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1732,"num_write_kb":13852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1730}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'1982","reported_seq":2057,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.851653+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:16.851653+0000","last_peered":"2024-09-10T20:20:16.851653+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:16.851653+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1982,"log_dups_size":0,"ondisk_log_size":1982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16210512,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":15838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":17}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1720","reported_seq":1771,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.039326+0000","last_change":"2024-09-10T20:19:53.542132+0000","last_active":"2024-09-10T20:20:09.039326+0000","last_peered":"2024-09-10T20:20:09.039326+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:19:31.154604+0000","last_became_peered":"2024-09-10T20:19:31.154604+0000","last_unstale":"2024-09-10T20:20:09.039326+0000","last_undegraded":"2024-09-10T20:19:31.144137+0000","last_fullsized":"2024-09-10T20:19:31.142952+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:54.145139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":13733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'1985","reported_seq":2026,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.447183+0000","last_change":"2024-09-10T20:20:13.450855+0000","last_active":"2024-09-10T20:20:13.450855+0000","last_peered":"2024-09-10T20:20:13.447183+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:19:31.154652+0000","last_became_peered":"2024-09-10T20:19:31.154652+0000","last_unstale":"2024-09-10T20:20:13.447183+0000","last_undegraded":"2024-09-10T20:19:31.147348+0000","last_fullsized":"2024-09-10T20:19:31.147057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1985,"log_dups_size":0,"ondisk_log_size":1985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233632,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":15861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'1938","reported_seq":1970,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.437352+0000","last_change":"2024-09-10T20:20:13.451390+0000","last_active":"2024-09-10T20:20:13.451390+0000","last_peered":"2024-09-10T20:20:13.437352+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:19:31.156062+0000","last_became_peered":"2024-09-10T20:19:31.156062+0000","last_unstale":"2024-09-10T20:20:13.437352+0000","last_undegraded":"2024-09-10T20:19:41.791102+0000","last_fullsized":"2024-09-10T20:19:31.147166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1938,"log_dups_size":0,"ondisk_log_size":1938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1938,"num_write_kb":15504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'1961","reported_seq":2015,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.416166+0000","last_change":"2024-09-10T20:20:13.447557+0000","last_active":"2024-09-10T20:20:13.447557+0000","last_peered":"2024-09-10T20:20:13.416166+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:19:31.155101+0000","last_became_peered":"2024-09-10T20:19:31.155101+0000","last_unstale":"2024-09-10T20:20:13.416166+0000","last_undegraded":"2024-09-10T20:19:31.146566+0000","last_fullsized":"2024-09-10T20:19:31.146192+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":15652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1779","reported_seq":1829,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.041843+0000","last_change":"2024-09-10T20:19:55.088547+0000","last_active":"2024-09-10T20:20:09.041843+0000","last_peered":"2024-09-10T20:20:09.041843+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:19:31.156208+0000","last_became_peered":"2024-09-10T20:19:31.156208+0000","last_unstale":"2024-09-10T20:20:09.041843+0000","last_undegraded":"2024-09-10T20:19:31.146143+0000","last_fullsized":"2024-09-10T20:19:31.145598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:56.214349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1774}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'1932","reported_seq":1977,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.446851+0000","last_change":"2024-09-10T20:20:13.454122+0000","last_active":"2024-09-10T20:20:13.454122+0000","last_peered":"2024-09-10T20:20:13.446851+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:19:31.154204+0000","last_became_peered":"2024-09-10T20:19:31.154204+0000","last_unstale":"2024-09-10T20:20:13.446851+0000","last_undegraded":"2024-09-10T20:19:31.147561+0000","last_fullsized":"2024-09-10T20:19:31.147205+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1932,"log_dups_size":0,"ondisk_log_size":1932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15755584,"num_objects":1924,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1929,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'1941","reported_seq":2016,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.907941+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:16.907941+0000","last_peered":"2024-09-10T20:20:16.907941+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:16.907941+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15881376,"num_objects":1939,"num_object_clones":0,"num_object_copies":7756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":15510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":36}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'1964","reported_seq":2008,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-10T20:20:13.442423+0000","last_change":"2024-09-10T20:20:13.454675+0000","last_active":"2024-09-10T20:20:13.454675+0000","last_peered":"2024-09-10T20:20:13.442423+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:19:31.153021+0000","last_became_peered":"2024-09-10T20:19:31.153021+0000","last_unstale":"2024-09-10T20:20:13.442423+0000","last_undegraded":"2024-09-10T20:19:31.145240+0000","last_fullsized":"2024-09-10T20:19:31.144893+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16043760,"num_objects":1959,"num_object_clones":0,"num_object_copies":7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":15676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1729","reported_seq":1759,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.128660+0000","last_change":"2024-09-10T20:19:57.121670+0000","last_active":"2024-09-10T20:20:09.128660+0000","last_peered":"2024-09-10T20:20:09.128660+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:19:31.154982+0000","last_became_peered":"2024-09-10T20:19:31.154982+0000","last_unstale":"2024-09-10T20:20:09.128660+0000","last_undegraded":"2024-09-10T20:19:31.148209+0000","last_fullsized":"2024-09-10T20:19:31.147914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:57.876065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14154320,"num_objects":1728,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'1977","reported_seq":2051,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.797668+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:16.797668+0000","last_peered":"2024-09-10T20:20:16.797668+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:16.797668+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16185936,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":15807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1763","reported_seq":1802,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.124320+0000","last_change":"2024-09-10T20:19:58.151835+0000","last_active":"2024-09-10T20:20:09.124320+0000","last_peered":"2024-09-10T20:20:09.124320+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:19:31.155530+0000","last_became_peered":"2024-09-10T20:19:31.155530+0000","last_unstale":"2024-09-10T20:20:09.124320+0000","last_undegraded":"2024-09-10T20:19:31.148111+0000","last_fullsized":"2024-09-10T20:19:31.147870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:20:58.907128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406816,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":14084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2005","reported_seq":2083,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.929371+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:16.929371+0000","last_peered":"2024-09-10T20:20:16.929371+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:16.929371+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2005,"log_dups_size":0,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16397472,"num_objects":2002,"num_object_clones":0,"num_object_copies":8008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":40}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1746","reported_seq":1796,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-10T20:20:09.005805+0000","last_change":"2024-09-10T20:19:59.268306+0000","last_active":"2024-09-10T20:20:09.005805+0000","last_peered":"2024-09-10T20:20:09.005805+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:19:31.155468+0000","last_became_peered":"2024-09-10T20:19:31.155468+0000","last_unstale":"2024-09-10T20:20:09.005805+0000","last_undegraded":"2024-09-10T20:19:31.145015+0000","last_fullsized":"2024-09-10T20:19:31.144654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1746,"log_dups_size":0,"ondisk_log_size":1746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-10T20:21:00.218351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259360,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1744,"num_write_kb":13940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":246276288,"num_objects":30068,"num_object_clones":0,"num_object_copies":120272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":240642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":391163904,"data_stored":391163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30122,"ondisk_log_size":30122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238296,"kb_used_data":126080,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94133544,"statfs":{"total":96636764160,"available":96392749056,"internally_reserved":0,"allocated":129105920,"data_stored":128938413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238820,"kb_used_data":127120,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94133020,"statfs":{"total":96636764160,"available":96392212480,"internally_reserved":0,"allocated":130170880,"data_stored":129994899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48324,"kb_used_data":2736,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94323516,"statfs":{"total":96636764160,"available":96587280384,"internally_reserved":0,"allocated":2801664,"data_stored":2625683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233088,"kb_used_data":125104,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":94138752,"statfs":{"total":96636764160,"available":96398082048,"internally_reserved":0,"allocated":128106496,"data_stored":127943568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":110554831},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2957312,"data_stored":2957312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130502656,"data_stored":130502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128851968,"data_stored":128851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:19.646 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:19.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 38 20 33230 33210 6.82687 3.49219 0.0309925 0.0228684 2024-09-10T20:20:20.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:20.207 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:20.235 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-10T20:20:19.889924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":267894976,"num_objects":32707,"num_object_clones":0,"num_object_copies":130828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32734,"num_write_kb":261754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32761,"ondisk_log_size":32761,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":773408,"kb_used_data":391768,"kb_used_omap":14,"kb_used_meta":381553,"kb_avail":376713952,"statfs":{"total":386547056640,"available":385755086848,"internally_reserved":0,"allocated":401170432,"data_stored":400466508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":390710838},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,12,13,5,0,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37429248,"num_objects":4569,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":36552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.464559"},"pg_stats":[{"pgid":"1.f","version":"32'2058","reported_seq":2099,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.029226+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:19.029226+0000","last_peered":"2024-09-10T20:20:19.029226+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:19.029226+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16859136,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":16464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":92}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2083","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.025584+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:19.025584+0000","last_peered":"2024-09-10T20:20:19.025584+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:19.025584+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028256,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":16644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'1982","reported_seq":2057,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.851653+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:16.851653+0000","last_peered":"2024-09-10T20:20:16.851653+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:16.851653+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1982,"log_dups_size":0,"ondisk_log_size":1982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16210512,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":15838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":17}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2106","reported_seq":2168,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.018558+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:19.018558+0000","last_peered":"2024-09-10T20:20:19.018558+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:19.018558+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17216672,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2104,"num_write_kb":16821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2094","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.118589+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:19.118589+0000","last_peered":"2024-09-10T20:20:19.118589+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:19.118589+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2094,"log_dups_size":0,"ondisk_log_size":2094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126560,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2092,"num_write_kb":16733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2020","reported_seq":2061,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.053183+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:19.053183+0000","last_peered":"2024-09-10T20:20:19.053183+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:19.053183+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2020,"log_dups_size":0,"ondisk_log_size":2020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16547840,"num_objects":2020,"num_object_clones":0,"num_object_copies":8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":16160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":82}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2048","reported_seq":2112,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.987485+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:18.987485+0000","last_peered":"2024-09-10T20:20:18.987485+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:18.987485+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2048,"log_dups_size":0,"ondisk_log_size":2048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16725152,"num_objects":2042,"num_object_clones":0,"num_object_copies":8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":16348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2124","reported_seq":2187,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.979541+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:18.979541+0000","last_peered":"2024-09-10T20:20:18.979541+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:18.979541+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17355936,"num_objects":2119,"num_object_clones":0,"num_object_copies":8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":16964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2032","reported_seq":2174,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.156049+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:19.156049+0000","last_peered":"2024-09-10T20:20:19.156049+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:19.156049+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2032,"log_dups_size":0,"ondisk_log_size":2032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16574784,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'1941","reported_seq":2016,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.907941+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:16.907941+0000","last_peered":"2024-09-10T20:20:16.907941+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:16.907941+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15881376,"num_objects":1939,"num_object_clones":0,"num_object_copies":7756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":15510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":36}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2052","reported_seq":2106,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.147169+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:19.147169+0000","last_peered":"2024-09-10T20:20:19.147169+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:19.147169+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16764656,"num_objects":2047,"num_object_clones":0,"num_object_copies":8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":16380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2090","reported_seq":2133,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.157498+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:19.157498+0000","last_peered":"2024-09-10T20:20:19.157498+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:19.157498+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111632,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2089,"num_write_kb":16711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'1977","reported_seq":2051,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.797668+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:16.797668+0000","last_peered":"2024-09-10T20:20:16.797668+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:16.797668+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16185936,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":15807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2088","reported_seq":2137,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.155456+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:19.155456+0000","last_peered":"2024-09-10T20:20:19.155456+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:19.155456+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17069216,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2005","reported_seq":2083,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.929371+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:16.929371+0000","last_peered":"2024-09-10T20:20:16.929371+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:16.929371+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2005,"log_dups_size":0,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16397472,"num_objects":2002,"num_object_clones":0,"num_object_copies":8008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":40}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2061","reported_seq":2124,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.982298+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:18.982298+0000","last_peered":"2024-09-10T20:20:18.982298+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:18.982298+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2061,"log_dups_size":0,"ondisk_log_size":2061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":267894976,"num_objects":32707,"num_object_clones":0,"num_object_copies":130828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32734,"num_write_kb":261754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403353600,"data_stored":403353600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32761,"ondisk_log_size":32761,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243912,"kb_used_data":131704,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94127928,"statfs":{"total":96636764160,"available":96386998272,"internally_reserved":0,"allocated":134864896,"data_stored":134688915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238820,"kb_used_data":127120,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94133020,"statfs":{"total":96636764160,"available":96392212480,"internally_reserved":0,"allocated":130170880,"data_stored":129994899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48324,"kb_used_data":2736,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94323516,"statfs":{"total":96636764160,"available":96587280384,"internally_reserved":0,"allocated":2801664,"data_stored":2625683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242352,"kb_used_data":130208,"kb_used_omap":4,"kb_used_meta":112123,"kb_avail":94129488,"statfs":{"total":96636764160,"available":96388595712,"internally_reserved":0,"allocated":133332992,"data_stored":133157011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114814671},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134832128,"data_stored":134832128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2957312,"data_stored":2957312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130502656,"data_stored":130502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135061504,"data_stored":135061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:20.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2058", 'reported_seq': 2099, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.029226+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:19.029226+0000', 'last_peered': '2024-09-10T20:20:19.029226+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:19.029226+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2058, 'log_dups_size': 0, 'ondisk_log_size': 2058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16859136, 'num_objects': 2058, 'num_object_clones': 0, 'num_object_copies': 8232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2058, 'num_write_kb': 16464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 92}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2083", 'reported_seq': 2145, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.025584+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:19.025584+0000', 'last_peered': '2024-09-10T20:20:19.025584+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:19.025584+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2083, 'log_dups_size': 0, 'ondisk_log_size': 2083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17028256, 'num_objects': 2079, 'num_object_clones': 0, 'num_object_copies': 8316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2081, 'num_write_kb': 16644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 88}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'1982", 'reported_seq': 2057, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:16.851653+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:16.851653+0000', 'last_peered': '2024-09-10T20:20:16.851653+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:16.851653+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1982, 'log_dups_size': 0, 'ondisk_log_size': 1982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16210512, 'num_objects': 1979, 'num_object_clones': 0, 'num_object_copies': 7916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1981, 'num_write_kb': 15838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 17}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2106", 'reported_seq': 2168, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.018558+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:19.018558+0000', 'last_peered': '2024-09-10T20:20:19.018558+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:19.018558+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2106, 'log_dups_size': 0, 'ondisk_log_size': 2106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17216672, 'num_objects': 2102, 'num_object_clones': 0, 'num_object_copies': 8408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2104, 'num_write_kb': 16821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 99}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2094", 'reported_seq': 2145, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.118589+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:19.118589+0000', 'last_peered': '2024-09-10T20:20:19.118589+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:19.118589+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2094, 'log_dups_size': 0, 'ondisk_log_size': 2094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17126560, 'num_objects': 2091, 'num_object_clones': 0, 'num_object_copies': 8364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2092, 'num_write_kb': 16733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 111}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2020", 'reported_seq': 2061, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.053183+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:19.053183+0000', 'last_peered': '2024-09-10T20:20:19.053183+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:19.053183+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2020, 'log_dups_size': 0, 'ondisk_log_size': 2020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16547840, 'num_objects': 2020, 'num_object_clones': 0, 'num_object_copies': 8080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2020, 'num_write_kb': 16160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 82}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2048", 'reported_seq': 2112, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:18.987485+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:18.987485+0000', 'last_peered': '2024-09-10T20:20:18.987485+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:18.987485+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2048, 'log_dups_size': 0, 'ondisk_log_size': 2048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16725152, 'num_objects': 2042, 'num_object_clones': 0, 'num_object_copies': 8168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2046, 'num_write_kb': 16348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 89}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2124", 'reported_seq': 2187, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:18.979541+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:18.979541+0000', 'last_peered': '2024-09-10T20:20:18.979541+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:18.979541+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2124, 'log_dups_size': 0, 'ondisk_log_size': 2124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17355936, 'num_objects': 2119, 'num_object_clones': 0, 'num_object_copies': 8476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2122, 'num_write_kb': 16964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 76}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2032", 'reported_seq': 2174, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.156049+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:19.156049+0000', 'last_peered': '2024-09-10T20:20:19.156049+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:19.156049+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2032, 'log_dups_size': 0, 'ondisk_log_size': 2032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16574784, 'num_objects': 2024, 'num_object_clones': 0, 'num_object_copies': 8096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2029, 'num_write_kb': 16209, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 352256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 147}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'1941", 'reported_seq': 2016, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:16.907941+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:16.907941+0000', 'last_peered': '2024-09-10T20:20:16.907941+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:16.907941+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1941, 'log_dups_size': 0, 'ondisk_log_size': 1941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15881376, 'num_objects': 1939, 'num_object_clones': 0, 'num_object_copies': 7756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1939, 'num_write_kb': 15510, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 36}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:20.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2052", 'reported_seq': 2106, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.147169+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:19.147169+0000', 'last_peered': '2024-09-10T20:20:19.147169+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:19.147169+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2052, 'log_dups_size': 0, 'ondisk_log_size': 2052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16764656, 'num_objects': 2047, 'num_object_clones': 0, 'num_object_copies': 8188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2049, 'num_write_kb': 16380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 91}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2090", 'reported_seq': 2133, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.157498+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:19.157498+0000', 'last_peered': '2024-09-10T20:20:19.157498+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:19.157498+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2090, 'log_dups_size': 0, 'ondisk_log_size': 2090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17111632, 'num_objects': 2089, 'num_object_clones': 0, 'num_object_copies': 8356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2089, 'num_write_kb': 16711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'1977", 'reported_seq': 2051, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:16.797668+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:16.797668+0000', 'last_peered': '2024-09-10T20:20:16.797668+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:16.797668+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 1977, 'log_dups_size': 0, 'ondisk_log_size': 1977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16185936, 'num_objects': 1976, 'num_object_clones': 0, 'num_object_copies': 7904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1976, 'num_write_kb': 15807, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 28}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2088", 'reported_seq': 2137, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.155456+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:19.155456+0000', 'last_peered': '2024-09-10T20:20:19.155456+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:19.155456+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2088, 'log_dups_size': 0, 'ondisk_log_size': 2088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17069216, 'num_objects': 2084, 'num_object_clones': 0, 'num_object_copies': 8336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2086, 'num_write_kb': 16684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2005", 'reported_seq': 2083, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:16.929371+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:16.929371+0000', 'last_peered': '2024-09-10T20:20:16.929371+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:16.929371+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2005, 'log_dups_size': 0, 'ondisk_log_size': 2005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16397472, 'num_objects': 2002, 'num_object_clones': 0, 'num_object_copies': 8008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2003, 'num_write_kb': 16021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 40}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:20.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2061", 'reported_seq': 2124, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:18.982298+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:18.982298+0000', 'last_peered': '2024-09-10T20:20:18.982298+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:18.982298+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2061, 'log_dups_size': 0, 'ondisk_log_size': 2061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16839840, 'num_objects': 2056, 'num_object_clones': 0, 'num_object_copies': 8224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2059, 'num_write_kb': 16460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 77}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:20.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:20.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 39 20 33651 33631 6.73615 3.28906 0.0429905 0.0231819 2024-09-10T20:20:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:20.936 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31438/130828 objects degraded (24.030%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":109,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32707,"data_bytes":267894976,"bytes_used":791969792,"bytes_avail":385755086848,"bytes_total":386547056640,"degraded_objects":31438,"degraded_total":130828,"degraded_ratio":0.24030024153850857,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":54490,"recovering_keys_per_sec":0,"num_objects_recovered":43,"num_bytes_recovered":352256,"num_keys_recovered":0,"write_bytes_sec":5789914,"read_op_per_sec":0,"write_op_per_sec":706},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-10T20:20:19.890029+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:20.964 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:20.965 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:21.535 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:21.536 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:21.560 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-10T20:20:19.889924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":267894976,"num_objects":32707,"num_object_clones":0,"num_object_copies":130828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32734,"num_write_kb":261754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32761,"ondisk_log_size":32761,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":773408,"kb_used_data":391768,"kb_used_omap":14,"kb_used_meta":381553,"kb_avail":376713952,"statfs":{"total":386547056640,"available":385755086848,"internally_reserved":0,"allocated":401170432,"data_stored":400466508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":390710838},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,12,13,5,0,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37429248,"num_objects":4569,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":36552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.464559"},"pg_stats":[{"pgid":"1.f","version":"32'2058","reported_seq":2099,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.029226+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:19.029226+0000","last_peered":"2024-09-10T20:20:19.029226+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:19.029226+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16859136,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":16464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":92}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2083","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.025584+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:19.025584+0000","last_peered":"2024-09-10T20:20:19.025584+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:19.025584+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028256,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":16644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'1982","reported_seq":2057,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.851653+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:16.851653+0000","last_peered":"2024-09-10T20:20:16.851653+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:16.851653+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1982,"log_dups_size":0,"ondisk_log_size":1982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16210512,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":15838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":17}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2106","reported_seq":2168,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.018558+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:19.018558+0000","last_peered":"2024-09-10T20:20:19.018558+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:19.018558+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17216672,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2104,"num_write_kb":16821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2094","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.118589+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:19.118589+0000","last_peered":"2024-09-10T20:20:19.118589+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:19.118589+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2094,"log_dups_size":0,"ondisk_log_size":2094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126560,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2092,"num_write_kb":16733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2020","reported_seq":2061,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.053183+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:19.053183+0000","last_peered":"2024-09-10T20:20:19.053183+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:19.053183+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2020,"log_dups_size":0,"ondisk_log_size":2020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16547840,"num_objects":2020,"num_object_clones":0,"num_object_copies":8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":16160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":82}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2048","reported_seq":2112,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.987485+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:18.987485+0000","last_peered":"2024-09-10T20:20:18.987485+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:18.987485+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2048,"log_dups_size":0,"ondisk_log_size":2048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16725152,"num_objects":2042,"num_object_clones":0,"num_object_copies":8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":16348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2124","reported_seq":2187,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.979541+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:18.979541+0000","last_peered":"2024-09-10T20:20:18.979541+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:18.979541+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17355936,"num_objects":2119,"num_object_clones":0,"num_object_copies":8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":16964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2032","reported_seq":2174,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.156049+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:19.156049+0000","last_peered":"2024-09-10T20:20:19.156049+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:19.156049+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2032,"log_dups_size":0,"ondisk_log_size":2032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16574784,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'1941","reported_seq":2016,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.907941+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:16.907941+0000","last_peered":"2024-09-10T20:20:16.907941+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:16.907941+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15881376,"num_objects":1939,"num_object_clones":0,"num_object_copies":7756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":15510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":36}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2052","reported_seq":2106,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.147169+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:19.147169+0000","last_peered":"2024-09-10T20:20:19.147169+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:19.147169+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16764656,"num_objects":2047,"num_object_clones":0,"num_object_copies":8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":16380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2090","reported_seq":2133,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.157498+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:19.157498+0000","last_peered":"2024-09-10T20:20:19.157498+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:19.157498+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111632,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2089,"num_write_kb":16711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'1977","reported_seq":2051,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.797668+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:16.797668+0000","last_peered":"2024-09-10T20:20:16.797668+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:16.797668+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16185936,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":15807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2088","reported_seq":2137,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.155456+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:19.155456+0000","last_peered":"2024-09-10T20:20:19.155456+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:19.155456+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17069216,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2005","reported_seq":2083,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:16.929371+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:16.929371+0000","last_peered":"2024-09-10T20:20:16.929371+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:16.929371+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2005,"log_dups_size":0,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16397472,"num_objects":2002,"num_object_clones":0,"num_object_copies":8008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":40}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2061","reported_seq":2124,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.982298+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:18.982298+0000","last_peered":"2024-09-10T20:20:18.982298+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:18.982298+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2061,"log_dups_size":0,"ondisk_log_size":2061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":267894976,"num_objects":32707,"num_object_clones":0,"num_object_copies":130828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32734,"num_write_kb":261754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403353600,"data_stored":403353600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32761,"ondisk_log_size":32761,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243912,"kb_used_data":131704,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94127928,"statfs":{"total":96636764160,"available":96386998272,"internally_reserved":0,"allocated":134864896,"data_stored":134688915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238820,"kb_used_data":127120,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94133020,"statfs":{"total":96636764160,"available":96392212480,"internally_reserved":0,"allocated":130170880,"data_stored":129994899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48324,"kb_used_data":2736,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94323516,"statfs":{"total":96636764160,"available":96587280384,"internally_reserved":0,"allocated":2801664,"data_stored":2625683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242352,"kb_used_data":130208,"kb_used_omap":4,"kb_used_meta":112123,"kb_avail":94129488,"statfs":{"total":96636764160,"available":96388595712,"internally_reserved":0,"allocated":133332992,"data_stored":133157011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114814671},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134832128,"data_stored":134832128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2957312,"data_stored":2957312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130502656,"data_stored":130502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135061504,"data_stored":135061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:21.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:20:21.771836+0000 min lat: 0.00230373 max lat: 2.76361 avg lat: 0.0234713 2024-09-10T20:20:21.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:20:21.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 40 20 34083 34063 6.65211 3.375 0.0219776 0.0234713 2024-09-10T20:20:22.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 41 20 34573 34553 6.58323 3.82812 0.0382591 0.0237218 2024-09-10T20:20:23.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 42 20 35060 35040 6.517 3.80469 0.0405987 0.0239577 2024-09-10T20:20:24.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:24.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 43 20 35362 35342 6.42031 2.35938 0.206564 0.024176 2024-09-10T20:20:24.863 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:24.863 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:24.875 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-10T20:20:23.890849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272613568,"num_objects":33283,"num_object_clones":0,"num_object_copies":133132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33310,"num_write_kb":266362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33337,"ondisk_log_size":33337,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":783792,"kb_used_data":410480,"kb_used_omap":14,"kb_used_meta":373233,"kb_avail":376703568,"statfs":{"total":386547056640,"available":385744453632,"internally_reserved":0,"allocated":420331520,"data_stored":419627596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":382191158},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,6,12,1,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29941760,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":29240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.453038"},"pg_stats":[{"pgid":"1.f","version":"32'2058","reported_seq":2099,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.029226+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:19.029226+0000","last_peered":"2024-09-10T20:20:19.029226+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:19.029226+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16859136,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":16464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":92}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2083","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.025584+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:19.025584+0000","last_peered":"2024-09-10T20:20:19.025584+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:19.025584+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028256,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":16644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2137","reported_seq":2212,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.887489+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:21.887489+0000","last_peered":"2024-09-10T20:20:21.887489+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:21.887489+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17480272,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2136,"num_write_kb":17078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":172}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2106","reported_seq":2168,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.018558+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:19.018558+0000","last_peered":"2024-09-10T20:20:19.018558+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:19.018558+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17216672,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2104,"num_write_kb":16821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2094","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.118589+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:19.118589+0000","last_peered":"2024-09-10T20:20:19.118589+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:19.118589+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2094,"log_dups_size":0,"ondisk_log_size":2094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126560,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2092,"num_write_kb":16733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2020","reported_seq":2061,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.053183+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:19.053183+0000","last_peered":"2024-09-10T20:20:19.053183+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:19.053183+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2020,"log_dups_size":0,"ondisk_log_size":2020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16547840,"num_objects":2020,"num_object_clones":0,"num_object_copies":8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":16160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":82}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2048","reported_seq":2112,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.987485+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:18.987485+0000","last_peered":"2024-09-10T20:20:18.987485+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:18.987485+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2048,"log_dups_size":0,"ondisk_log_size":2048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16725152,"num_objects":2042,"num_object_clones":0,"num_object_copies":8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":16348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2124","reported_seq":2187,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.979541+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:18.979541+0000","last_peered":"2024-09-10T20:20:18.979541+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:18.979541+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17355936,"num_objects":2119,"num_object_clones":0,"num_object_copies":8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":16964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2032","reported_seq":2174,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.156049+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:19.156049+0000","last_peered":"2024-09-10T20:20:19.156049+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:19.156049+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2032,"log_dups_size":0,"ondisk_log_size":2032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16574784,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2083","reported_seq":2159,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.951789+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:21.951789+0000","last_peered":"2024-09-10T20:20:21.951789+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:21.951789+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044640,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":16646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2052","reported_seq":2106,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.147169+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:19.147169+0000","last_peered":"2024-09-10T20:20:19.147169+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:19.147169+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16764656,"num_objects":2047,"num_object_clones":0,"num_object_copies":8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":16380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2090","reported_seq":2133,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.157498+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:19.157498+0000","last_peered":"2024-09-10T20:20:19.157498+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:19.157498+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111632,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2089,"num_write_kb":16711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2103","reported_seq":2177,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.889811+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:21.889811+0000","last_peered":"2024-09-10T20:20:21.889811+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:21.889811+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17218128,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2102,"num_write_kb":16815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2088","reported_seq":2137,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.155456+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:19.155456+0000","last_peered":"2024-09-10T20:20:19.155456+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:19.155456+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17069216,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2158","reported_seq":2234,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.951624+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:21.951624+0000","last_peered":"2024-09-10T20:20:21.951624+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:21.951624+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17650848,"num_objects":2155,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2156,"num_write_kb":17245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2061","reported_seq":2124,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.982298+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:18.982298+0000","last_peered":"2024-09-10T20:20:18.982298+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:18.982298+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2061,"log_dups_size":0,"ondisk_log_size":2061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":272613568,"num_objects":33283,"num_object_clones":0,"num_object_copies":133132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33310,"num_write_kb":266362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":422486016,"data_stored":422486016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33337,"ondisk_log_size":33337,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243912,"kb_used_data":131704,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94127928,"statfs":{"total":96636764160,"available":96386998272,"internally_reserved":0,"allocated":134864896,"data_stored":134688915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235636,"kb_used_data":136420,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94136204,"statfs":{"total":96636764160,"available":96395472896,"internally_reserved":0,"allocated":139694080,"data_stored":139518099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61892,"kb_used_data":12148,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94309948,"statfs":{"total":96636764160,"available":96573386752,"internally_reserved":0,"allocated":12439552,"data_stored":12263571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242352,"kb_used_data":130208,"kb_used_omap":4,"kb_used_meta":112123,"kb_avail":94129488,"statfs":{"total":96636764160,"available":96388595712,"internally_reserved":0,"allocated":133332992,"data_stored":133157011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114814671},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134832128,"data_stored":134832128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12632064,"data_stored":12632064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139960320,"data_stored":139960320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135061504,"data_stored":135061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:24.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:25.224 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:25.225 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:25.237 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-10T20:20:23.890849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272613568,"num_objects":33283,"num_object_clones":0,"num_object_copies":133132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33310,"num_write_kb":266362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33337,"ondisk_log_size":33337,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":783792,"kb_used_data":410480,"kb_used_omap":14,"kb_used_meta":373233,"kb_avail":376703568,"statfs":{"total":386547056640,"available":385744453632,"internally_reserved":0,"allocated":420331520,"data_stored":419627596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":382191158},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,6,12,1,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29941760,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":29240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.453038"},"pg_stats":[{"pgid":"1.f","version":"32'2058","reported_seq":2099,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.029226+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:19.029226+0000","last_peered":"2024-09-10T20:20:19.029226+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:19.029226+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16859136,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":16464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":92}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2083","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.025584+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:19.025584+0000","last_peered":"2024-09-10T20:20:19.025584+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:19.025584+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028256,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":16644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2137","reported_seq":2212,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.887489+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:21.887489+0000","last_peered":"2024-09-10T20:20:21.887489+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:21.887489+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17480272,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2136,"num_write_kb":17078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":172}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2106","reported_seq":2168,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.018558+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:19.018558+0000","last_peered":"2024-09-10T20:20:19.018558+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:19.018558+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17216672,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2104,"num_write_kb":16821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2094","reported_seq":2145,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.118589+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:19.118589+0000","last_peered":"2024-09-10T20:20:19.118589+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:19.118589+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2094,"log_dups_size":0,"ondisk_log_size":2094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126560,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2092,"num_write_kb":16733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2020","reported_seq":2061,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.053183+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:19.053183+0000","last_peered":"2024-09-10T20:20:19.053183+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:19.053183+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2020,"log_dups_size":0,"ondisk_log_size":2020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16547840,"num_objects":2020,"num_object_clones":0,"num_object_copies":8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":16160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":82}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2048","reported_seq":2112,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.987485+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:18.987485+0000","last_peered":"2024-09-10T20:20:18.987485+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:18.987485+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2048,"log_dups_size":0,"ondisk_log_size":2048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16725152,"num_objects":2042,"num_object_clones":0,"num_object_copies":8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":16348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2124","reported_seq":2187,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.979541+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:18.979541+0000","last_peered":"2024-09-10T20:20:18.979541+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:18.979541+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17355936,"num_objects":2119,"num_object_clones":0,"num_object_copies":8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":16964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2032","reported_seq":2174,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.156049+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:19.156049+0000","last_peered":"2024-09-10T20:20:19.156049+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:19.156049+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2032,"log_dups_size":0,"ondisk_log_size":2032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16574784,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2083","reported_seq":2159,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.951789+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:21.951789+0000","last_peered":"2024-09-10T20:20:21.951789+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:21.951789+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044640,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":16646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2052","reported_seq":2106,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.147169+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:19.147169+0000","last_peered":"2024-09-10T20:20:19.147169+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:19.147169+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16764656,"num_objects":2047,"num_object_clones":0,"num_object_copies":8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":16380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2090","reported_seq":2133,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.157498+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:19.157498+0000","last_peered":"2024-09-10T20:20:19.157498+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:19.157498+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111632,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2089,"num_write_kb":16711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2103","reported_seq":2177,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.889811+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:21.889811+0000","last_peered":"2024-09-10T20:20:21.889811+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:21.889811+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17218128,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2102,"num_write_kb":16815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2088","reported_seq":2137,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:19.155456+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:19.155456+0000","last_peered":"2024-09-10T20:20:19.155456+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:19.155456+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17069216,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2158","reported_seq":2234,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.951624+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:21.951624+0000","last_peered":"2024-09-10T20:20:21.951624+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:21.951624+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17650848,"num_objects":2155,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2156,"num_write_kb":17245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2061","reported_seq":2124,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:18.982298+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:18.982298+0000","last_peered":"2024-09-10T20:20:18.982298+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:18.982298+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2061,"log_dups_size":0,"ondisk_log_size":2061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":272613568,"num_objects":33283,"num_object_clones":0,"num_object_copies":133132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33310,"num_write_kb":266362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":422486016,"data_stored":422486016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33337,"ondisk_log_size":33337,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243912,"kb_used_data":131704,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94127928,"statfs":{"total":96636764160,"available":96386998272,"internally_reserved":0,"allocated":134864896,"data_stored":134688915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":114880208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,4,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235636,"kb_used_data":136420,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94136204,"statfs":{"total":96636764160,"available":96395472896,"internally_reserved":0,"allocated":139694080,"data_stored":139518099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61892,"kb_used_data":12148,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94309948,"statfs":{"total":96636764160,"available":96573386752,"internally_reserved":0,"allocated":12439552,"data_stored":12263571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242352,"kb_used_data":130208,"kb_used_omap":4,"kb_used_meta":112123,"kb_avail":94129488,"statfs":{"total":96636764160,"available":96388595712,"internally_reserved":0,"allocated":133332992,"data_stored":133157011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114814671},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134832128,"data_stored":134832128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12632064,"data_stored":12632064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139960320,"data_stored":139960320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135061504,"data_stored":135061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:25.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:25.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2058", 'reported_seq': 2099, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.029226+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:19.029226+0000', 'last_peered': '2024-09-10T20:20:19.029226+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:19.029226+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2058, 'log_dups_size': 0, 'ondisk_log_size': 2058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16859136, 'num_objects': 2058, 'num_object_clones': 0, 'num_object_copies': 8232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2058, 'num_write_kb': 16464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 92}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:25.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:25.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2083", 'reported_seq': 2145, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.025584+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:19.025584+0000', 'last_peered': '2024-09-10T20:20:19.025584+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:19.025584+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2083, 'log_dups_size': 0, 'ondisk_log_size': 2083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17028256, 'num_objects': 2079, 'num_object_clones': 0, 'num_object_copies': 8316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2081, 'num_write_kb': 16644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 88}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:25.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:25.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2137", 'reported_seq': 2212, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:21.887489+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:21.887489+0000', 'last_peered': '2024-09-10T20:20:21.887489+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:21.887489+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2137, 'log_dups_size': 0, 'ondisk_log_size': 2137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17480272, 'num_objects': 2134, 'num_object_clones': 0, 'num_object_copies': 8536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2136, 'num_write_kb': 17078, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 172}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2106", 'reported_seq': 2168, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.018558+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:19.018558+0000', 'last_peered': '2024-09-10T20:20:19.018558+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:19.018558+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2106, 'log_dups_size': 0, 'ondisk_log_size': 2106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17216672, 'num_objects': 2102, 'num_object_clones': 0, 'num_object_copies': 8408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2104, 'num_write_kb': 16821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 99}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2094", 'reported_seq': 2145, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.118589+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:19.118589+0000', 'last_peered': '2024-09-10T20:20:19.118589+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:19.118589+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2094, 'log_dups_size': 0, 'ondisk_log_size': 2094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17126560, 'num_objects': 2091, 'num_object_clones': 0, 'num_object_copies': 8364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2092, 'num_write_kb': 16733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 111}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2020", 'reported_seq': 2061, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.053183+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:19.053183+0000', 'last_peered': '2024-09-10T20:20:19.053183+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:19.053183+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2020, 'log_dups_size': 0, 'ondisk_log_size': 2020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16547840, 'num_objects': 2020, 'num_object_clones': 0, 'num_object_copies': 8080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2020, 'num_write_kb': 16160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 82}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2048", 'reported_seq': 2112, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:18.987485+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:18.987485+0000', 'last_peered': '2024-09-10T20:20:18.987485+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:18.987485+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2048, 'log_dups_size': 0, 'ondisk_log_size': 2048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16725152, 'num_objects': 2042, 'num_object_clones': 0, 'num_object_copies': 8168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2046, 'num_write_kb': 16348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 89}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2124", 'reported_seq': 2187, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:18.979541+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:18.979541+0000', 'last_peered': '2024-09-10T20:20:18.979541+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:18.979541+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2124, 'log_dups_size': 0, 'ondisk_log_size': 2124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17355936, 'num_objects': 2119, 'num_object_clones': 0, 'num_object_copies': 8476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2122, 'num_write_kb': 16964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 76}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:25.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2032", 'reported_seq': 2174, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.156049+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:19.156049+0000', 'last_peered': '2024-09-10T20:20:19.156049+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:19.156049+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2032, 'log_dups_size': 0, 'ondisk_log_size': 2032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16574784, 'num_objects': 2024, 'num_object_clones': 0, 'num_object_copies': 8096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2029, 'num_write_kb': 16209, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 352256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 147}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2083", 'reported_seq': 2159, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:21.951789+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:21.951789+0000', 'last_peered': '2024-09-10T20:20:21.951789+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:21.951789+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2083, 'log_dups_size': 0, 'ondisk_log_size': 2083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17044640, 'num_objects': 2081, 'num_object_clones': 0, 'num_object_copies': 8324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2081, 'num_write_kb': 16646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 178}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2052", 'reported_seq': 2106, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.147169+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:19.147169+0000', 'last_peered': '2024-09-10T20:20:19.147169+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:19.147169+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2052, 'log_dups_size': 0, 'ondisk_log_size': 2052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16764656, 'num_objects': 2047, 'num_object_clones': 0, 'num_object_copies': 8188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2049, 'num_write_kb': 16380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 91}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2090", 'reported_seq': 2133, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.157498+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:19.157498+0000', 'last_peered': '2024-09-10T20:20:19.157498+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:19.157498+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2090, 'log_dups_size': 0, 'ondisk_log_size': 2090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17111632, 'num_objects': 2089, 'num_object_clones': 0, 'num_object_copies': 8356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2089, 'num_write_kb': 16711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2103", 'reported_seq': 2177, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:21.889811+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:21.889811+0000', 'last_peered': '2024-09-10T20:20:21.889811+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:21.889811+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2103, 'log_dups_size': 0, 'ondisk_log_size': 2103, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17218128, 'num_objects': 2102, 'num_object_clones': 0, 'num_object_copies': 8408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2102, 'num_write_kb': 16815, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:25.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:25.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2088", 'reported_seq': 2137, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:19.155456+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:19.155456+0000', 'last_peered': '2024-09-10T20:20:19.155456+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:19.155456+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2088, 'log_dups_size': 0, 'ondisk_log_size': 2088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17069216, 'num_objects': 2084, 'num_object_clones': 0, 'num_object_copies': 8336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2086, 'num_write_kb': 16684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:25.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:25.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2158", 'reported_seq': 2234, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:21.951624+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:21.951624+0000', 'last_peered': '2024-09-10T20:20:21.951624+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:21.951624+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2158, 'log_dups_size': 0, 'ondisk_log_size': 2158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17650848, 'num_objects': 2155, 'num_object_clones': 0, 'num_object_copies': 8620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2156, 'num_write_kb': 17245, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 193}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:25.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:25.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2061", 'reported_seq': 2124, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:18.982298+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:18.982298+0000', 'last_peered': '2024-09-10T20:20:18.982298+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:18.982298+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2061, 'log_dups_size': 0, 'ondisk_log_size': 2061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16839840, 'num_objects': 2056, 'num_object_clones': 0, 'num_object_copies': 8224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2059, 'num_write_kb': 16460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 77}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:25.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:25.636 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:25.636 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31438/133132 objects degraded (23.614%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":114,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33283,"data_bytes":272613568,"bytes_used":802603008,"bytes_avail":385744453632,"bytes_total":386547056640,"degraded_objects":31438,"degraded_total":133132,"degraded_ratio":0.23614157377640241,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":37263,"recovering_keys_per_sec":0,"num_objects_recovered":43,"num_bytes_recovered":352256,"num_keys_recovered":0,"write_bytes_sec":3167421,"read_op_per_sec":0,"write_op_per_sec":386},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-10T20:20:19.890029+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:25.649 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:25.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:25.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 44 20 35362 35342 6.27441 0 - 0.024176 2024-09-10T20:20:26.043 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:26.044 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:26.066 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-10T20:20:25.891842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286712000,"num_objects":35004,"num_object_clones":0,"num_object_copies":140016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35031,"num_write_kb":280130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35058,"ondisk_log_size":35058,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":817168,"kb_used_data":428560,"kb_used_omap":14,"kb_used_meta":388529,"kb_avail":376670192,"statfs":{"total":386547056640,"available":385710276608,"internally_reserved":0,"allocated":438845440,"data_stored":438141516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":397854262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,9,16,16,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44040192,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":43008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.443517"},"pg_stats":[{"pgid":"1.f","version":"32'2204","reported_seq":2245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.135694+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:24.135694+0000","last_peered":"2024-09-10T20:20:24.135694+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:24.135694+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2204,"log_dups_size":0,"ondisk_log_size":2204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18055168,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2204,"num_write_kb":17632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2221","reported_seq":2283,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:23.953239+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:23.953239+0000","last_peered":"2024-09-10T20:20:23.953239+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:23.953239+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2221,"log_dups_size":0,"ondisk_log_size":2221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18158752,"num_objects":2217,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2219,"num_write_kb":17748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2137","reported_seq":2212,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.887489+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:21.887489+0000","last_peered":"2024-09-10T20:20:21.887489+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:21.887489+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17480272,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2136,"num_write_kb":17078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":172}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2243","reported_seq":2304,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.001820+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:24.001820+0000","last_peered":"2024-09-10T20:20:24.001820+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:24.001820+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18338976,"num_objects":2239,"num_object_clones":0,"num_object_copies":8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2251","reported_seq":2302,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.079392+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:24.079392+0000","last_peered":"2024-09-10T20:20:24.079392+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:24.079392+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18412704,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2173","reported_seq":2214,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.184076+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:24.184076+0000","last_peered":"2024-09-10T20:20:24.184076+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:24.184076+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17801216,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":17384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2191","reported_seq":2255,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.017777+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:24.017777+0000","last_peered":"2024-09-10T20:20:24.017777+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:24.017777+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2191,"log_dups_size":0,"ondisk_log_size":2191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896608,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2189,"num_write_kb":17492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":232}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2254","reported_seq":2317,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:23.981300+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:23.981300+0000","last_peered":"2024-09-10T20:20:23.981300+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:23.981300+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2254,"log_dups_size":0,"ondisk_log_size":2254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420896,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2252,"num_write_kb":18004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2174","reported_seq":2444,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.189926+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:24.189926+0000","last_peered":"2024-09-10T20:20:24.189926+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:24.189926+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17738048,"num_objects":2166,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":17345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2083","reported_seq":2159,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.951789+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:21.951789+0000","last_peered":"2024-09-10T20:20:21.951789+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:21.951789+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044640,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":16646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2207","reported_seq":2261,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.151574+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:24.151574+0000","last_peered":"2024-09-10T20:20:24.151574+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:24.151574+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18034416,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2204,"num_write_kb":17620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2221","reported_seq":2264,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.128395+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:24.128395+0000","last_peered":"2024-09-10T20:20:24.128395+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:24.128395+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2221,"log_dups_size":0,"ondisk_log_size":2221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18184784,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2220,"num_write_kb":17759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2103","reported_seq":2177,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.889811+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:21.889811+0000","last_peered":"2024-09-10T20:20:21.889811+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:21.889811+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17218128,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2102,"num_write_kb":16815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2231","reported_seq":2281,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.129318+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:24.129318+0000","last_peered":"2024-09-10T20:20:24.129318+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:24.129318+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18240672,"num_objects":2227,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":17828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2158","reported_seq":2234,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:21.951624+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:21.951624+0000","last_peered":"2024-09-10T20:20:21.951624+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:21.951624+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17650848,"num_objects":2155,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2156,"num_write_kb":17245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2207","reported_seq":2270,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:23.940805+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:23.940805+0000","last_peered":"2024-09-10T20:20:23.940805+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:23.940805+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18035872,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2205,"num_write_kb":17628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":286712000,"num_objects":35004,"num_object_clones":0,"num_object_copies":140016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35031,"num_write_kb":280130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":441147392,"data_stored":441147392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35058,"ondisk_log_size":35058,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":262684,"kb_used_data":140748,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94109156,"statfs":{"total":96636764160,"available":96367775744,"internally_reserved":0,"allocated":144125952,"data_stored":143949971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":124841680},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,7,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235636,"kb_used_data":136420,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94136204,"statfs":{"total":96636764160,"available":96395472896,"internally_reserved":0,"allocated":139694080,"data_stored":139518099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61892,"kb_used_data":12148,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94309948,"statfs":{"total":96636764160,"available":96573386752,"internally_reserved":0,"allocated":12439552,"data_stored":12263571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256956,"kb_used_data":139244,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94114884,"statfs":{"total":96636764160,"available":96373641216,"internally_reserved":0,"allocated":142585856,"data_stored":142409875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":120516303},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":144171008,"data_stored":144171008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12632064,"data_stored":12632064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139960320,"data_stored":139960320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144384000,"data_stored":144384000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:26.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 45 20 35795 35775 6.21015 1.69141 0.0295861 0.0251453 2024-09-10T20:20:27.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 46 20 36202 36182 6.14426 3.17969 0.0413164 0.0254116 2024-09-10T20:20:28.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 47 20 36679 36659 6.09282 3.72656 0.0418206 0.0256268 2024-09-10T20:20:29.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:29.619 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:29.619 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:29.630 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-10T20:20:27.892323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289972416,"num_objects":35402,"num_object_clones":0,"num_object_copies":141608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35429,"num_write_kb":283314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35456,"ondisk_log_size":35456,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":862420,"kb_used_data":443284,"kb_used_omap":14,"kb_used_meta":419057,"kb_avail":376624940,"statfs":{"total":386547056640,"available":385663938560,"internally_reserved":0,"allocated":453922816,"data_stored":453218892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":429114934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,6,11,13,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47300608,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":46192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004817"},"pg_stats":[{"pgid":"1.f","version":"32'2204","reported_seq":2245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.135694+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:24.135694+0000","last_peered":"2024-09-10T20:20:24.135694+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:24.135694+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2204,"log_dups_size":0,"ondisk_log_size":2204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18055168,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2204,"num_write_kb":17632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2221","reported_seq":2283,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:23.953239+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:23.953239+0000","last_peered":"2024-09-10T20:20:23.953239+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:23.953239+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2221,"log_dups_size":0,"ondisk_log_size":2221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18158752,"num_objects":2217,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2219,"num_write_kb":17748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2232","reported_seq":2307,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.927704+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:26.927704+0000","last_peered":"2024-09-10T20:20:26.927704+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:26.927704+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2232,"log_dups_size":0,"ondisk_log_size":2232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18258512,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2231,"num_write_kb":17838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":267}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2243","reported_seq":2304,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.001820+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:24.001820+0000","last_peered":"2024-09-10T20:20:24.001820+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:24.001820+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18338976,"num_objects":2239,"num_object_clones":0,"num_object_copies":8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2251","reported_seq":2302,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.079392+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:24.079392+0000","last_peered":"2024-09-10T20:20:24.079392+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:24.079392+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18412704,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2173","reported_seq":2214,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.184076+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:24.184076+0000","last_peered":"2024-09-10T20:20:24.184076+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:24.184076+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17801216,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":17384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2191","reported_seq":2255,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.017777+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:24.017777+0000","last_peered":"2024-09-10T20:20:24.017777+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:24.017777+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2191,"log_dups_size":0,"ondisk_log_size":2191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896608,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2189,"num_write_kb":17492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":232}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2254","reported_seq":2317,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:23.981300+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:23.981300+0000","last_peered":"2024-09-10T20:20:23.981300+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:23.981300+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2254,"log_dups_size":0,"ondisk_log_size":2254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420896,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2252,"num_write_kb":18004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2174","reported_seq":2444,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.189926+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:24.189926+0000","last_peered":"2024-09-10T20:20:24.189926+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:24.189926+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17738048,"num_objects":2166,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":17345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2181","reported_seq":2255,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.934624+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:26.934624+0000","last_peered":"2024-09-10T20:20:26.934624+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:26.934624+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17847456,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2179,"num_write_kb":17430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":276}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2207","reported_seq":2261,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.151574+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:24.151574+0000","last_peered":"2024-09-10T20:20:24.151574+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:24.151574+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18034416,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2204,"num_write_kb":17620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2221","reported_seq":2264,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.128395+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:24.128395+0000","last_peered":"2024-09-10T20:20:24.128395+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:24.128395+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2221,"log_dups_size":0,"ondisk_log_size":2221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18184784,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2220,"num_write_kb":17759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2202","reported_seq":2276,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.877390+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:26.877390+0000","last_peered":"2024-09-10T20:20:26.877390+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:26.877390+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18029136,"num_objects":2201,"num_object_clones":0,"num_object_copies":8804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":17607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":253}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2231","reported_seq":2281,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:24.129318+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:24.129318+0000","last_peered":"2024-09-10T20:20:24.129318+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:24.129318+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18240672,"num_objects":2227,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":17828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2264","reported_seq":2342,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.880756+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:26.880756+0000","last_peered":"2024-09-10T20:20:26.880756+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:26.880756+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2264,"log_dups_size":0,"ondisk_log_size":2264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18519200,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":18093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2207","reported_seq":2270,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:23.940805+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:23.940805+0000","last_peered":"2024-09-10T20:20:23.940805+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:23.940805+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18035872,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2205,"num_write_kb":17628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":289972416,"num_objects":35402,"num_object_clones":0,"num_object_copies":141608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35429,"num_write_kb":283314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":456146944,"data_stored":456146944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35456,"ondisk_log_size":35456,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":262684,"kb_used_data":140748,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94109156,"statfs":{"total":96636764160,"available":96367775744,"internally_reserved":0,"allocated":144125952,"data_stored":143949971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":124841680},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,7,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264720,"kb_used_data":143296,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94107120,"statfs":{"total":96636764160,"available":96365690880,"internally_reserved":0,"allocated":146735104,"data_stored":146559123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78060,"kb_used_data":19996,"kb_used_omap":4,"kb_used_meta":58043,"kb_avail":94293780,"statfs":{"total":96636764160,"available":96556830720,"internally_reserved":0,"allocated":20475904,"data_stored":20299923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":59436754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256956,"kb_used_data":139244,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94114884,"statfs":{"total":96636764160,"available":96373641216,"internally_reserved":0,"allocated":142585856,"data_stored":142409875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":120516303},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":144171008,"data_stored":144171008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20623360,"data_stored":20623360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146968576,"data_stored":146968576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144384000,"data_stored":144384000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:29.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:29.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 48 20 37101 37081 6.03457 3.29688 0.036139 0.0258735 2024-09-10T20:20:30.253 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:30.277 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-10T20:20:29.893286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":299983040,"num_objects":36624,"num_object_clones":0,"num_object_copies":146496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36651,"num_write_kb":293090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36678,"ondisk_log_size":36678,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":901784,"kb_used_data":456276,"kb_used_omap":14,"kb_used_meta":445425,"kb_avail":376585576,"statfs":{"total":386547056640,"available":385623629824,"internally_reserved":0,"allocated":467226624,"data_stored":466522700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":456115766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,14,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":53706752,"num_objects":6556,"num_object_clones":0,"num_object_copies":26224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":52448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003769"},"pg_stats":[{"pgid":"1.f","version":"32'2300","reported_seq":2340,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.180610+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:29.180610+0000","last_peered":"2024-09-10T20:20:29.180610+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:29.180610+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2300,"log_dups_size":0,"ondisk_log_size":2300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18841600,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":18400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2332","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.035914+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:29.035914+0000","last_peered":"2024-09-10T20:20:29.035914+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:29.035914+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19068064,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2232","reported_seq":2307,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.927704+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:26.927704+0000","last_peered":"2024-09-10T20:20:26.927704+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:26.927704+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2232,"log_dups_size":0,"ondisk_log_size":2232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18258512,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2231,"num_write_kb":17838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":267}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2355","reported_seq":2417,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.006455+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:29.006455+0000","last_peered":"2024-09-10T20:20:29.006455+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:29.006455+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2355,"log_dups_size":0,"ondisk_log_size":2355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19256480,"num_objects":2351,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":18813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2344","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.087582+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:29.087582+0000","last_peered":"2024-09-10T20:20:29.087582+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:29.087582+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2342,"num_write_kb":18733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2282","reported_seq":2323,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.187025+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:29.187025+0000","last_peered":"2024-09-10T20:20:29.187025+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:29.187025+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18694144,"num_objects":2282,"num_object_clones":0,"num_object_copies":9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":18256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2296","reported_seq":2360,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:28.963311+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:28.963311+0000","last_peered":"2024-09-10T20:20:28.963311+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:28.963311+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756768,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2294,"num_write_kb":18332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":337}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2345","reported_seq":2408,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.039150+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:29.039150+0000","last_peered":"2024-09-10T20:20:29.039150+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:29.039150+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19166368,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2259","reported_seq":3025,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.144468+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:29.144468+0000","last_peered":"2024-09-10T20:20:29.144468+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:29.144468+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18434368,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":18025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2181","reported_seq":2255,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.934624+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:26.934624+0000","last_peered":"2024-09-10T20:20:26.934624+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:26.934624+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17847456,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2179,"num_write_kb":17430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":276}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2315","reported_seq":2369,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.116584+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:29.116584+0000","last_peered":"2024-09-10T20:20:29.116584+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:29.116584+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18919152,"num_objects":2310,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":18484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2339","reported_seq":2382,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.197035+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:29.197035+0000","last_peered":"2024-09-10T20:20:29.197035+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:29.197035+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19151440,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2202","reported_seq":2276,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.877390+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:26.877390+0000","last_peered":"2024-09-10T20:20:26.877390+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:26.877390+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18029136,"num_objects":2201,"num_object_clones":0,"num_object_copies":8804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":17607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":253}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2322","reported_seq":2372,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.126439+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:29.126439+0000","last_peered":"2024-09-10T20:20:29.126439+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:29.126439+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18986144,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":343}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2264","reported_seq":2342,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.880756+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:26.880756+0000","last_peered":"2024-09-10T20:20:26.880756+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:26.880756+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2264,"log_dups_size":0,"ondisk_log_size":2264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18519200,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":18093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2310","reported_seq":2373,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.033814+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:29.033814+0000","last_peered":"2024-09-10T20:20:29.033814+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:29.033814+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18879648,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2308,"num_write_kb":18452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":299983040,"num_objects":36624,"num_object_clones":0,"num_object_copies":146496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36651,"num_write_kb":293090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":469544960,"data_stored":469544960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36678,"ondisk_log_size":36678,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291428,"kb_used_data":147284,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94080412,"statfs":{"total":96636764160,"available":96338341888,"internally_reserved":0,"allocated":150818816,"data_stored":150642835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264720,"kb_used_data":143296,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94107120,"statfs":{"total":96636764160,"available":96365690880,"internally_reserved":0,"allocated":146735104,"data_stored":146559123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78060,"kb_used_data":19996,"kb_used_omap":4,"kb_used_meta":58043,"kb_avail":94293780,"statfs":{"total":96636764160,"available":96556830720,"internally_reserved":0,"allocated":20475904,"data_stored":20299923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":59436754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267576,"kb_used_data":145700,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94104264,"statfs":{"total":96636764160,"available":96362766336,"internally_reserved":0,"allocated":149196800,"data_stored":149020819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150847488,"data_stored":150847488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20623360,"data_stored":20623360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146968576,"data_stored":146968576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151105536,"data_stored":151105536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:30.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:30.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2300", 'reported_seq': 2340, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.180610+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:29.180610+0000', 'last_peered': '2024-09-10T20:20:29.180610+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:29.180610+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2300, 'log_dups_size': 0, 'ondisk_log_size': 2300, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18841600, 'num_objects': 2300, 'num_object_clones': 0, 'num_object_copies': 9200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2300, 'num_write_kb': 18400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 334}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:30.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:30.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2332", 'reported_seq': 2395, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.035914+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:29.035914+0000', 'last_peered': '2024-09-10T20:20:29.035914+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:29.035914+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2332, 'log_dups_size': 0, 'ondisk_log_size': 2332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19068064, 'num_objects': 2328, 'num_object_clones': 0, 'num_object_copies': 9312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2330, 'num_write_kb': 18636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2232", 'reported_seq': 2307, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:26.927704+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:26.927704+0000', 'last_peered': '2024-09-10T20:20:26.927704+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:26.927704+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2232, 'log_dups_size': 0, 'ondisk_log_size': 2232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18258512, 'num_objects': 2229, 'num_object_clones': 0, 'num_object_copies': 8916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2231, 'num_write_kb': 17838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 267}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2355", 'reported_seq': 2417, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.006455+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:29.006455+0000', 'last_peered': '2024-09-10T20:20:29.006455+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:29.006455+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2355, 'log_dups_size': 0, 'ondisk_log_size': 2355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19256480, 'num_objects': 2351, 'num_object_clones': 0, 'num_object_copies': 9404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2353, 'num_write_kb': 18813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2344", 'reported_seq': 2395, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.087582+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:29.087582+0000', 'last_peered': '2024-09-10T20:20:29.087582+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:29.087582+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2344, 'log_dups_size': 0, 'ondisk_log_size': 2344, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19174560, 'num_objects': 2341, 'num_object_clones': 0, 'num_object_copies': 9364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2342, 'num_write_kb': 18733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 361}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:30.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2282", 'reported_seq': 2323, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.187025+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:29.187025+0000', 'last_peered': '2024-09-10T20:20:29.187025+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:29.187025+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2282, 'log_dups_size': 0, 'ondisk_log_size': 2282, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18694144, 'num_objects': 2282, 'num_object_clones': 0, 'num_object_copies': 9128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2282, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2282, 'num_write_kb': 18256, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 344}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2296", 'reported_seq': 2360, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:28.963311+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:28.963311+0000', 'last_peered': '2024-09-10T20:20:28.963311+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:28.963311+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2296, 'log_dups_size': 0, 'ondisk_log_size': 2296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18756768, 'num_objects': 2290, 'num_object_clones': 0, 'num_object_copies': 9160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2294, 'num_write_kb': 18332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 337}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2345", 'reported_seq': 2408, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.039150+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:29.039150+0000', 'last_peered': '2024-09-10T20:20:29.039150+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:29.039150+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2345, 'log_dups_size': 0, 'ondisk_log_size': 2345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19166368, 'num_objects': 2340, 'num_object_clones': 0, 'num_object_copies': 9360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2343, 'num_write_kb': 18732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2259", 'reported_seq': 3025, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.144468+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:29.144468+0000', 'last_peered': '2024-09-10T20:20:29.144468+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:29.144468+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2259, 'log_dups_size': 0, 'ondisk_log_size': 2259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18434368, 'num_objects': 2251, 'num_object_clones': 0, 'num_object_copies': 9004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1565, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2256, 'num_write_kb': 18025, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 358, 'num_bytes_recovered': 2908160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 686}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2181", 'reported_seq': 2255, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:26.934624+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:26.934624+0000', 'last_peered': '2024-09-10T20:20:26.934624+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:26.934624+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2181, 'log_dups_size': 0, 'ondisk_log_size': 2181, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17847456, 'num_objects': 2179, 'num_object_clones': 0, 'num_object_copies': 8716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2179, 'num_write_kb': 17430, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 276}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:30.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2315", 'reported_seq': 2369, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.116584+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:29.116584+0000', 'last_peered': '2024-09-10T20:20:29.116584+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:29.116584+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2315, 'log_dups_size': 0, 'ondisk_log_size': 2315, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18919152, 'num_objects': 2310, 'num_object_clones': 0, 'num_object_copies': 9240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2312, 'num_write_kb': 18484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 354}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2339", 'reported_seq': 2382, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.197035+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:29.197035+0000', 'last_peered': '2024-09-10T20:20:29.197035+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:29.197035+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2339, 'log_dups_size': 0, 'ondisk_log_size': 2339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19151440, 'num_objects': 2338, 'num_object_clones': 0, 'num_object_copies': 9352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2338, 'num_write_kb': 18703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 360}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2202", 'reported_seq': 2276, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:26.877390+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:26.877390+0000', 'last_peered': '2024-09-10T20:20:26.877390+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:26.877390+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2202, 'log_dups_size': 0, 'ondisk_log_size': 2202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18029136, 'num_objects': 2201, 'num_object_clones': 0, 'num_object_copies': 8804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2201, 'num_write_kb': 17607, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 253}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2322", 'reported_seq': 2372, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.126439+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:29.126439+0000', 'last_peered': '2024-09-10T20:20:29.126439+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:29.126439+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2322, 'log_dups_size': 0, 'ondisk_log_size': 2322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18986144, 'num_objects': 2318, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2320, 'num_write_kb': 18556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2264", 'reported_seq': 2342, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:26.880756+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:26.880756+0000', 'last_peered': '2024-09-10T20:20:26.880756+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:26.880756+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2264, 'log_dups_size': 0, 'ondisk_log_size': 2264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18519200, 'num_objects': 2261, 'num_object_clones': 0, 'num_object_copies': 9044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2262, 'num_write_kb': 18093, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:30.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2310", 'reported_seq': 2373, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.033814+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:29.033814+0000', 'last_peered': '2024-09-10T20:20:29.033814+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:29.033814+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2310, 'log_dups_size': 0, 'ondisk_log_size': 2310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18879648, 'num_objects': 2305, 'num_object_clones': 0, 'num_object_copies': 9220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2308, 'num_write_kb': 18452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:30.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:30.775 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 49 20 37522 37502 5.97853 3.28906 0.0433513 0.026119 2024-09-10T20:20:31.001 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:31.001 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31126/146496 objects degraded (21.247%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36624,"data_bytes":299983040,"bytes_used":923426816,"bytes_avail":385623629824,"bytes_total":386547056640,"degraded_objects":31126,"degraded_total":146496,"degraded_ratio":0.21246996505024027,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":242270,"recovering_keys_per_sec":0,"num_objects_recovered":355,"num_bytes_recovered":2908160,"num_keys_recovered":0,"write_bytes_sec":4474157,"read_op_per_sec":0,"write_op_per_sec":546},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:31.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:31.650 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:31.651 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:31.680 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-10T20:20:29.893286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":299983040,"num_objects":36624,"num_object_clones":0,"num_object_copies":146496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36651,"num_write_kb":293090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36678,"ondisk_log_size":36678,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":901784,"kb_used_data":456276,"kb_used_omap":14,"kb_used_meta":445425,"kb_avail":376585576,"statfs":{"total":386547056640,"available":385623629824,"internally_reserved":0,"allocated":467226624,"data_stored":466522700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":456115766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,14,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":53706752,"num_objects":6556,"num_object_clones":0,"num_object_copies":26224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":52448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003769"},"pg_stats":[{"pgid":"1.f","version":"32'2300","reported_seq":2340,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.180610+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:29.180610+0000","last_peered":"2024-09-10T20:20:29.180610+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:29.180610+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2300,"log_dups_size":0,"ondisk_log_size":2300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18841600,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":18400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2332","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.035914+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:29.035914+0000","last_peered":"2024-09-10T20:20:29.035914+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:29.035914+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19068064,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2232","reported_seq":2307,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.927704+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:26.927704+0000","last_peered":"2024-09-10T20:20:26.927704+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:26.927704+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2232,"log_dups_size":0,"ondisk_log_size":2232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18258512,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2231,"num_write_kb":17838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":267}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2355","reported_seq":2417,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.006455+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:29.006455+0000","last_peered":"2024-09-10T20:20:29.006455+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:29.006455+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2355,"log_dups_size":0,"ondisk_log_size":2355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19256480,"num_objects":2351,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":18813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2344","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.087582+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:29.087582+0000","last_peered":"2024-09-10T20:20:29.087582+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:29.087582+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2342,"num_write_kb":18733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2282","reported_seq":2323,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.187025+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:29.187025+0000","last_peered":"2024-09-10T20:20:29.187025+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:29.187025+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18694144,"num_objects":2282,"num_object_clones":0,"num_object_copies":9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":18256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2296","reported_seq":2360,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:28.963311+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:28.963311+0000","last_peered":"2024-09-10T20:20:28.963311+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:28.963311+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756768,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2294,"num_write_kb":18332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":337}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2345","reported_seq":2408,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.039150+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:29.039150+0000","last_peered":"2024-09-10T20:20:29.039150+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:29.039150+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19166368,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2259","reported_seq":3025,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.144468+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:29.144468+0000","last_peered":"2024-09-10T20:20:29.144468+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:29.144468+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18434368,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":18025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2181","reported_seq":2255,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.934624+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:26.934624+0000","last_peered":"2024-09-10T20:20:26.934624+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:26.934624+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17847456,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2179,"num_write_kb":17430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":276}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2315","reported_seq":2369,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.116584+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:29.116584+0000","last_peered":"2024-09-10T20:20:29.116584+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:29.116584+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18919152,"num_objects":2310,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":18484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2339","reported_seq":2382,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.197035+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:29.197035+0000","last_peered":"2024-09-10T20:20:29.197035+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:29.197035+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19151440,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2202","reported_seq":2276,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.877390+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:26.877390+0000","last_peered":"2024-09-10T20:20:26.877390+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:26.877390+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18029136,"num_objects":2201,"num_object_clones":0,"num_object_copies":8804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":17607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":253}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2322","reported_seq":2372,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.126439+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:29.126439+0000","last_peered":"2024-09-10T20:20:29.126439+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:29.126439+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18986144,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":343}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2264","reported_seq":2342,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:26.880756+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:26.880756+0000","last_peered":"2024-09-10T20:20:26.880756+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:26.880756+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2264,"log_dups_size":0,"ondisk_log_size":2264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18519200,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":18093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2310","reported_seq":2373,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.033814+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:29.033814+0000","last_peered":"2024-09-10T20:20:29.033814+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:29.033814+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18879648,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2308,"num_write_kb":18452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":299983040,"num_objects":36624,"num_object_clones":0,"num_object_copies":146496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36651,"num_write_kb":293090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":469544960,"data_stored":469544960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36678,"ondisk_log_size":36678,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291428,"kb_used_data":147284,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94080412,"statfs":{"total":96636764160,"available":96338341888,"internally_reserved":0,"allocated":150818816,"data_stored":150642835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264720,"kb_used_data":143296,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94107120,"statfs":{"total":96636764160,"available":96365690880,"internally_reserved":0,"allocated":146735104,"data_stored":146559123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78060,"kb_used_data":19996,"kb_used_omap":4,"kb_used_meta":58043,"kb_avail":94293780,"statfs":{"total":96636764160,"available":96556830720,"internally_reserved":0,"allocated":20475904,"data_stored":20299923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":59436754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267576,"kb_used_data":145700,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94104264,"statfs":{"total":96636764160,"available":96362766336,"internally_reserved":0,"allocated":149196800,"data_stored":149020819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150847488,"data_stored":150847488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20623360,"data_stored":20623360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146968576,"data_stored":146968576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151105536,"data_stored":151105536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:31.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 50 20 37943 37923 5.92467 3.28906 0.042643 0.0263532 2024-09-10T20:20:32.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 51 20 38421 38401 5.88172 3.73438 0.0385529 0.0265463 2024-09-10T20:20:33.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 52 20 38898 38878 5.84025 3.72656 0.0878387 0.0267389 2024-09-10T20:20:34.682 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:34.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 53 20 39356 39336 5.79757 3.57812 0.0499135 0.0269305 2024-09-10T20:20:35.263 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:35.264 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:35.292 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-10T20:20:33.894215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":304693440,"num_objects":37199,"num_object_clones":0,"num_object_copies":148796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37226,"num_write_kb":297690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37253,"ondisk_log_size":37253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":933496,"kb_used_data":474100,"kb_used_omap":14,"kb_used_meta":459313,"kb_avail":376553864,"statfs":{"total":386547056640,"available":385591156736,"internally_reserved":0,"allocated":485478400,"data_stored":484774476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":470337078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,15,4,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36798464,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":2555904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003798"},"pg_stats":[{"pgid":"1.f","version":"32'2300","reported_seq":2340,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.180610+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:29.180610+0000","last_peered":"2024-09-10T20:20:29.180610+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:29.180610+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2300,"log_dups_size":0,"ondisk_log_size":2300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18841600,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":18400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2332","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.035914+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:29.035914+0000","last_peered":"2024-09-10T20:20:29.035914+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:29.035914+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19068064,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2363","reported_seq":2437,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.952489+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:31.952489+0000","last_peered":"2024-09-10T20:20:31.952489+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:31.952489+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2363,"log_dups_size":0,"ondisk_log_size":2363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19331664,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2355","reported_seq":2417,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.006455+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:29.006455+0000","last_peered":"2024-09-10T20:20:29.006455+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:29.006455+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2355,"log_dups_size":0,"ondisk_log_size":2355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19256480,"num_objects":2351,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":18813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2344","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.087582+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:29.087582+0000","last_peered":"2024-09-10T20:20:29.087582+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:29.087582+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2342,"num_write_kb":18733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2282","reported_seq":2323,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.187025+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:29.187025+0000","last_peered":"2024-09-10T20:20:29.187025+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:29.187025+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18694144,"num_objects":2282,"num_object_clones":0,"num_object_copies":9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":18256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2296","reported_seq":2360,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:28.963311+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:28.963311+0000","last_peered":"2024-09-10T20:20:28.963311+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:28.963311+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756768,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2294,"num_write_kb":18332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":337}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2345","reported_seq":2408,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.039150+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:29.039150+0000","last_peered":"2024-09-10T20:20:29.039150+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:29.039150+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19166368,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2259","reported_seq":3025,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.144468+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:29.144468+0000","last_peered":"2024-09-10T20:20:29.144468+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:29.144468+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18434368,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":18025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2322","reported_seq":2397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.985939+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:31.985939+0000","last_peered":"2024-09-10T20:20:31.985939+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:31.985939+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2315","reported_seq":2369,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.116584+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:29.116584+0000","last_peered":"2024-09-10T20:20:29.116584+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:29.116584+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18919152,"num_objects":2310,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":18484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2339","reported_seq":2382,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.197035+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:29.197035+0000","last_peered":"2024-09-10T20:20:29.197035+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:29.197035+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19151440,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2354","reported_seq":2428,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.959658+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:31.959658+0000","last_peered":"2024-09-10T20:20:31.959658+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:31.959658+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19274320,"num_objects":2353,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":18823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":405}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2322","reported_seq":2372,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.126439+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:29.126439+0000","last_peered":"2024-09-10T20:20:29.126439+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:29.126439+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18986144,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":343}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2415","reported_seq":2493,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.955971+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:31.955971+0000","last_peered":"2024-09-10T20:20:31.955971+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:31.955971+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19756192,"num_objects":2412,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2413,"num_write_kb":19301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2310","reported_seq":2373,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.033814+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:29.033814+0000","last_peered":"2024-09-10T20:20:29.033814+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:29.033814+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18879648,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2308,"num_write_kb":18452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":304693440,"num_objects":37199,"num_object_clones":0,"num_object_copies":148796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37226,"num_write_kb":297690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487563264,"data_stored":487563264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37253,"ondisk_log_size":37253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291428,"kb_used_data":147284,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94080412,"statfs":{"total":96636764160,"available":96338341888,"internally_reserved":0,"allocated":150818816,"data_stored":150642835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299908,"kb_used_data":152116,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94071932,"statfs":{"total":96636764160,"available":96329658368,"internally_reserved":0,"allocated":155766784,"data_stored":155590803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74584,"kb_used_data":29000,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94297256,"statfs":{"total":96636764160,"available":96560390144,"internally_reserved":0,"allocated":29696000,"data_stored":29520019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267576,"kb_used_data":145700,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94104264,"statfs":{"total":96636764160,"available":96362766336,"internally_reserved":0,"allocated":149196800,"data_stored":149020819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150847488,"data_stored":150847488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29745152,"data_stored":29745152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155865088,"data_stored":155865088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151105536,"data_stored":151105536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:35.293 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:35.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 54 20 39710 39690 5.74142 2.76562 0.0561786 0.0271954 2024-09-10T20:20:35.779 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:35.779 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:35.807 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-10T20:20:33.894215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":304693440,"num_objects":37199,"num_object_clones":0,"num_object_copies":148796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37226,"num_write_kb":297690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37253,"ondisk_log_size":37253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":933496,"kb_used_data":474100,"kb_used_omap":14,"kb_used_meta":459313,"kb_avail":376553864,"statfs":{"total":386547056640,"available":385591156736,"internally_reserved":0,"allocated":485478400,"data_stored":484774476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":470337078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,15,4,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36798464,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":2555904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003798"},"pg_stats":[{"pgid":"1.f","version":"32'2300","reported_seq":2340,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.180610+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:29.180610+0000","last_peered":"2024-09-10T20:20:29.180610+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:29.180610+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2300,"log_dups_size":0,"ondisk_log_size":2300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18841600,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":18400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2332","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.035914+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:29.035914+0000","last_peered":"2024-09-10T20:20:29.035914+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:29.035914+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19068064,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2363","reported_seq":2437,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.952489+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:31.952489+0000","last_peered":"2024-09-10T20:20:31.952489+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:31.952489+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2363,"log_dups_size":0,"ondisk_log_size":2363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19331664,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2355","reported_seq":2417,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.006455+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:29.006455+0000","last_peered":"2024-09-10T20:20:29.006455+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:29.006455+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2355,"log_dups_size":0,"ondisk_log_size":2355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19256480,"num_objects":2351,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":18813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2344","reported_seq":2395,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.087582+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:29.087582+0000","last_peered":"2024-09-10T20:20:29.087582+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:29.087582+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2342,"num_write_kb":18733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2282","reported_seq":2323,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.187025+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:29.187025+0000","last_peered":"2024-09-10T20:20:29.187025+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:29.187025+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18694144,"num_objects":2282,"num_object_clones":0,"num_object_copies":9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":18256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2296","reported_seq":2360,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:28.963311+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:28.963311+0000","last_peered":"2024-09-10T20:20:28.963311+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:28.963311+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756768,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2294,"num_write_kb":18332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":337}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2345","reported_seq":2408,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.039150+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:29.039150+0000","last_peered":"2024-09-10T20:20:29.039150+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:29.039150+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19166368,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2259","reported_seq":3025,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.144468+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:29.144468+0000","last_peered":"2024-09-10T20:20:29.144468+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:29.144468+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18434368,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":18025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2322","reported_seq":2397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.985939+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:31.985939+0000","last_peered":"2024-09-10T20:20:31.985939+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:31.985939+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2315","reported_seq":2369,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.116584+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:29.116584+0000","last_peered":"2024-09-10T20:20:29.116584+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:29.116584+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18919152,"num_objects":2310,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":18484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2339","reported_seq":2382,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.197035+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:29.197035+0000","last_peered":"2024-09-10T20:20:29.197035+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:29.197035+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19151440,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2354","reported_seq":2428,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.959658+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:31.959658+0000","last_peered":"2024-09-10T20:20:31.959658+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:31.959658+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19274320,"num_objects":2353,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":18823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":405}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2322","reported_seq":2372,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.126439+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:29.126439+0000","last_peered":"2024-09-10T20:20:29.126439+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:29.126439+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18986144,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":343}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2415","reported_seq":2493,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.955971+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:31.955971+0000","last_peered":"2024-09-10T20:20:31.955971+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:31.955971+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19756192,"num_objects":2412,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2413,"num_write_kb":19301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2310","reported_seq":2373,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:29.033814+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:29.033814+0000","last_peered":"2024-09-10T20:20:29.033814+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:29.033814+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18879648,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2308,"num_write_kb":18452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":304693440,"num_objects":37199,"num_object_clones":0,"num_object_copies":148796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37226,"num_write_kb":297690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":2908160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487563264,"data_stored":487563264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37253,"ondisk_log_size":37253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291428,"kb_used_data":147284,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94080412,"statfs":{"total":96636764160,"available":96338341888,"internally_reserved":0,"allocated":150818816,"data_stored":150642835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299908,"kb_used_data":152116,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94071932,"statfs":{"total":96636764160,"available":96329658368,"internally_reserved":0,"allocated":155766784,"data_stored":155590803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74584,"kb_used_data":29000,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94297256,"statfs":{"total":96636764160,"available":96560390144,"internally_reserved":0,"allocated":29696000,"data_stored":29520019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267576,"kb_used_data":145700,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94104264,"statfs":{"total":96636764160,"available":96362766336,"internally_reserved":0,"allocated":149196800,"data_stored":149020819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150847488,"data_stored":150847488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29745152,"data_stored":29745152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155865088,"data_stored":155865088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151105536,"data_stored":151105536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:35.808 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:35.808 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2300", 'reported_seq': 2340, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.180610+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:29.180610+0000', 'last_peered': '2024-09-10T20:20:29.180610+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:29.180610+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2300, 'log_dups_size': 0, 'ondisk_log_size': 2300, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18841600, 'num_objects': 2300, 'num_object_clones': 0, 'num_object_copies': 9200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2300, 'num_write_kb': 18400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 334}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2332", 'reported_seq': 2395, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.035914+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:29.035914+0000', 'last_peered': '2024-09-10T20:20:29.035914+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:29.035914+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2332, 'log_dups_size': 0, 'ondisk_log_size': 2332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19068064, 'num_objects': 2328, 'num_object_clones': 0, 'num_object_copies': 9312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2330, 'num_write_kb': 18636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2363", 'reported_seq': 2437, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:31.952489+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:31.952489+0000', 'last_peered': '2024-09-10T20:20:31.952489+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:31.952489+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2363, 'log_dups_size': 0, 'ondisk_log_size': 2363, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19331664, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2362, 'num_write_kb': 18886, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 398}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2355", 'reported_seq': 2417, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.006455+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:29.006455+0000', 'last_peered': '2024-09-10T20:20:29.006455+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:29.006455+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2355, 'log_dups_size': 0, 'ondisk_log_size': 2355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19256480, 'num_objects': 2351, 'num_object_clones': 0, 'num_object_copies': 9404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2353, 'num_write_kb': 18813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2344", 'reported_seq': 2395, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.087582+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:29.087582+0000', 'last_peered': '2024-09-10T20:20:29.087582+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:29.087582+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2344, 'log_dups_size': 0, 'ondisk_log_size': 2344, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19174560, 'num_objects': 2341, 'num_object_clones': 0, 'num_object_copies': 9364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2342, 'num_write_kb': 18733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 361}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2282", 'reported_seq': 2323, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.187025+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:29.187025+0000', 'last_peered': '2024-09-10T20:20:29.187025+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:29.187025+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2282, 'log_dups_size': 0, 'ondisk_log_size': 2282, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18694144, 'num_objects': 2282, 'num_object_clones': 0, 'num_object_copies': 9128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2282, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2282, 'num_write_kb': 18256, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 344}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:35.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2296", 'reported_seq': 2360, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:28.963311+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:28.963311+0000', 'last_peered': '2024-09-10T20:20:28.963311+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:28.963311+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2296, 'log_dups_size': 0, 'ondisk_log_size': 2296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18756768, 'num_objects': 2290, 'num_object_clones': 0, 'num_object_copies': 9160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2294, 'num_write_kb': 18332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 337}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2345", 'reported_seq': 2408, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.039150+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:29.039150+0000', 'last_peered': '2024-09-10T20:20:29.039150+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:29.039150+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2345, 'log_dups_size': 0, 'ondisk_log_size': 2345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19166368, 'num_objects': 2340, 'num_object_clones': 0, 'num_object_copies': 9360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2343, 'num_write_kb': 18732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2259", 'reported_seq': 3025, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.144468+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:29.144468+0000', 'last_peered': '2024-09-10T20:20:29.144468+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:29.144468+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2259, 'log_dups_size': 0, 'ondisk_log_size': 2259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18434368, 'num_objects': 2251, 'num_object_clones': 0, 'num_object_copies': 9004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1565, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2256, 'num_write_kb': 18025, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 358, 'num_bytes_recovered': 2908160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 686}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2322", 'reported_seq': 2397, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:31.985939+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:31.985939+0000', 'last_peered': '2024-09-10T20:20:31.985939+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:31.985939+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2322, 'log_dups_size': 0, 'ondisk_log_size': 2322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19002528, 'num_objects': 2320, 'num_object_clones': 0, 'num_object_copies': 9280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2320, 'num_write_kb': 18558, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 417}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2315", 'reported_seq': 2369, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.116584+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:29.116584+0000', 'last_peered': '2024-09-10T20:20:29.116584+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:29.116584+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2315, 'log_dups_size': 0, 'ondisk_log_size': 2315, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18919152, 'num_objects': 2310, 'num_object_clones': 0, 'num_object_copies': 9240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2312, 'num_write_kb': 18484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 354}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:35.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2339", 'reported_seq': 2382, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.197035+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:29.197035+0000', 'last_peered': '2024-09-10T20:20:29.197035+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:29.197035+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2339, 'log_dups_size': 0, 'ondisk_log_size': 2339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19151440, 'num_objects': 2338, 'num_object_clones': 0, 'num_object_copies': 9352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2338, 'num_write_kb': 18703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 360}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2354", 'reported_seq': 2428, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:31.959658+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:31.959658+0000', 'last_peered': '2024-09-10T20:20:31.959658+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:31.959658+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2354, 'log_dups_size': 0, 'ondisk_log_size': 2354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19274320, 'num_objects': 2353, 'num_object_clones': 0, 'num_object_copies': 9412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2353, 'num_write_kb': 18823, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2322", 'reported_seq': 2372, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.126439+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:29.126439+0000', 'last_peered': '2024-09-10T20:20:29.126439+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:29.126439+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2322, 'log_dups_size': 0, 'ondisk_log_size': 2322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18986144, 'num_objects': 2318, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2320, 'num_write_kb': 18556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2415", 'reported_seq': 2493, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:31.955971+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:31.955971+0000', 'last_peered': '2024-09-10T20:20:31.955971+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:31.955971+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2415, 'log_dups_size': 0, 'ondisk_log_size': 2415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19756192, 'num_objects': 2412, 'num_object_clones': 0, 'num_object_copies': 9648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2413, 'num_write_kb': 19301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:35.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2310", 'reported_seq': 2373, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:29.033814+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:29.033814+0000', 'last_peered': '2024-09-10T20:20:29.033814+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:29.033814+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2310, 'log_dups_size': 0, 'ondisk_log_size': 2310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18879648, 'num_objects': 2305, 'num_object_clones': 0, 'num_object_copies': 9220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2308, 'num_write_kb': 18452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:35.812 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:36.523 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:36.523 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31126/148796 objects degraded (20.919%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":125,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37199,"data_bytes":304693440,"bytes_used":955899904,"bytes_avail":385591156736,"bytes_total":386547056640,"degraded_objects":31126,"degraded_total":148796,"degraded_ratio":0.20918573079921504,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":212924,"recovering_keys_per_sec":0,"num_objects_recovered":312,"num_bytes_recovered":2555904,"num_keys_recovered":0,"write_bytes_sec":3065568,"read_op_per_sec":0,"write_op_per_sec":374},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:36.553 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:36.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:36.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 55 20 40112 40092 5.69413 3.14062 0.0176021 0.0274173 2024-09-10T20:20:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:36.986 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:37.007 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-10T20:20:35.897153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318120128,"num_objects":38838,"num_object_clones":0,"num_object_copies":155352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38865,"num_write_kb":310802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":3481600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38892,"ondisk_log_size":38892,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":977772,"kb_used_data":492012,"kb_used_omap":14,"kb_used_meta":485681,"kb_avail":376509588,"statfs":{"total":386547056640,"available":385545818112,"internally_reserved":0,"allocated":503820288,"data_stored":503116364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":497337910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,11,8,25,12,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45506560,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":44440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006304"},"pg_stats":[{"pgid":"1.f","version":"32'2429","reported_seq":2470,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.213691+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:34.213691+0000","last_peered":"2024-09-10T20:20:34.213691+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:34.213691+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2429,"log_dups_size":0,"ondisk_log_size":2429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19898368,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2496","reported_seq":2559,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.045293+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:34.045293+0000","last_peered":"2024-09-10T20:20:34.045293+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:34.045293+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2496,"log_dups_size":0,"ondisk_log_size":2496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20411552,"num_objects":2492,"num_object_clones":0,"num_object_copies":9968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2494,"num_write_kb":19948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2363","reported_seq":2437,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.952489+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:31.952489+0000","last_peered":"2024-09-10T20:20:31.952489+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:31.952489+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2363,"log_dups_size":0,"ondisk_log_size":2363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19331664,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2487","reported_seq":2549,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.090385+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:34.090385+0000","last_peered":"2024-09-10T20:20:34.090385+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:34.090385+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337824,"num_objects":2483,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2485,"num_write_kb":19869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":480}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2484","reported_seq":2535,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.201827+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:34.201827+0000","last_peered":"2024-09-10T20:20:34.201827+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:34.201827+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2484,"log_dups_size":0,"ondisk_log_size":2484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321440,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2419","reported_seq":2460,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.199145+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:34.199145+0000","last_peered":"2024-09-10T20:20:34.199145+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:34.199145+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2419,"log_dups_size":0,"ondisk_log_size":2419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19816448,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2419,"num_write_kb":19352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2442","reported_seq":2506,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.057515+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:34.057515+0000","last_peered":"2024-09-10T20:20:34.057515+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:34.057515+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2442,"log_dups_size":0,"ondisk_log_size":2442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952800,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2440,"num_write_kb":19500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":483}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2475","reported_seq":2538,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.059519+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:34.059519+0000","last_peered":"2024-09-10T20:20:34.059519+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:34.059519+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2475,"log_dups_size":0,"ondisk_log_size":2475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231328,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2473,"num_write_kb":19772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":427}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2401","reported_seq":3306,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.235695+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:34.235695+0000","last_peered":"2024-09-10T20:20:34.235695+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:34.235695+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2401,"log_dups_size":0,"ondisk_log_size":2401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19597632,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2398,"num_write_kb":19161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":428,"num_bytes_recovered":3481600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2322","reported_seq":2397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.985939+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:31.985939+0000","last_peered":"2024-09-10T20:20:31.985939+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:31.985939+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2432","reported_seq":2485,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.189675+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:34.189675+0000","last_peered":"2024-09-10T20:20:34.189675+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:34.189675+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2432,"log_dups_size":0,"ondisk_log_size":2432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19877616,"num_objects":2427,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":471}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2478","reported_seq":2521,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.229780+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:34.229780+0000","last_peered":"2024-09-10T20:20:34.229780+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:34.229780+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2478,"log_dups_size":0,"ondisk_log_size":2478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20290128,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":19815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":499}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2354","reported_seq":2428,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.959658+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:31.959658+0000","last_peered":"2024-09-10T20:20:31.959658+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:31.959658+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19274320,"num_objects":2353,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":18823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":405}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2454","reported_seq":2504,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.173406+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:34.173406+0000","last_peered":"2024-09-10T20:20:34.173406+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:34.173406+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":475}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2415","reported_seq":2493,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:31.955971+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:31.955971+0000","last_peered":"2024-09-10T20:20:31.955971+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:31.955971+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19756192,"num_objects":2412,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2413,"num_write_kb":19301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2441","reported_seq":2504,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:34.101926+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:34.101926+0000","last_peered":"2024-09-10T20:20:34.101926+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:34.101926+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952800,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":318120128,"num_objects":38838,"num_object_clones":0,"num_object_copies":155352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38865,"num_write_kb":310802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":3481600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":505917440,"data_stored":505917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38892,"ondisk_log_size":38892,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304420,"kb_used_data":156116,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94067420,"statfs":{"total":96636764160,"available":96325038080,"internally_reserved":0,"allocated":159862784,"data_stored":159686803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,4,8,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299908,"kb_used_data":152116,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94071932,"statfs":{"total":96636764160,"available":96329658368,"internally_reserved":0,"allocated":155766784,"data_stored":155590803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74584,"kb_used_data":29000,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94297256,"statfs":{"total":96636764160,"available":96560390144,"internally_reserved":0,"allocated":29696000,"data_stored":29520019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298860,"kb_used_data":154780,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94072980,"statfs":{"total":96636764160,"available":96330731520,"internally_reserved":0,"allocated":158494720,"data_stored":158318739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,11,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160026624,"data_stored":160026624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29745152,"data_stored":29745152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155865088,"data_stored":155865088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":160280576,"data_stored":160280576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:37.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 56 20 40426 40406 5.63625 2.45312 0.0403893 0.0277045 2024-09-10T20:20:38.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 57 20 40754 40734 5.58233 2.5625 0.00847488 0.027904 2024-09-10T20:20:39.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 58 20 41138 41118 5.5378 3 0.0398559 0.0282009 2024-09-10T20:20:40.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:40.624 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:40.625 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:40.648 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-10T20:20:39.898113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334012608,"num_objects":40778,"num_object_clones":0,"num_object_copies":163112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40805,"num_write_kb":326322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40832,"ondisk_log_size":40832,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":997600,"kb_used_data":524320,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376489760,"statfs":{"total":386547056640,"available":385525514240,"internally_reserved":0,"allocated":536903680,"data_stored":536199756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":484558390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,11,4,5,22,18,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44040192,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":43008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005789"},"pg_stats":[{"pgid":"1.f","version":"32'2544","reported_seq":2585,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.221678+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:39.221678+0000","last_peered":"2024-09-10T20:20:39.221678+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:39.221678+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2544,"log_dups_size":0,"ondisk_log_size":2544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2544,"num_write_kb":20352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2594","reported_seq":2656,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.100847+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:39.100847+0000","last_peered":"2024-09-10T20:20:39.100847+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:39.100847+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2594,"log_dups_size":0,"ondisk_log_size":2594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214368,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":20732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2488","reported_seq":2563,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.755154+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:36.755154+0000","last_peered":"2024-09-10T20:20:36.755154+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:36.755154+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2488,"log_dups_size":0,"ondisk_log_size":2488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20355664,"num_objects":2485,"num_object_clones":0,"num_object_copies":9940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":19886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2593","reported_seq":2655,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:38.972389+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:38.972389+0000","last_peered":"2024-09-10T20:20:38.972389+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:38.972389+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206176,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2591,"num_write_kb":20717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2610","reported_seq":2661,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.235585+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:39.235585+0000","last_peered":"2024-09-10T20:20:39.235585+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:39.235585+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2610,"log_dups_size":0,"ondisk_log_size":2610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21353632,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2550","reported_seq":2591,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.127912+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:39.127912+0000","last_peered":"2024-09-10T20:20:39.127912+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:39.127912+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20889600,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":20400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2561","reported_seq":2625,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.132213+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:39.132213+0000","last_peered":"2024-09-10T20:20:39.132213+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:39.132213+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20927648,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2577","reported_seq":2640,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.092539+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:39.092539+0000","last_peered":"2024-09-10T20:20:39.092539+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:39.092539+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2577,"log_dups_size":0,"ondisk_log_size":2577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21066912,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2575,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2536","reported_seq":3720,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.300114+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:39.300114+0000","last_peered":"2024-09-10T20:20:39.300114+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:39.300114+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20703552,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":20241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2460","reported_seq":2536,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.969096+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:36.969096+0000","last_peered":"2024-09-10T20:20:36.969096+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:36.969096+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2460,"log_dups_size":0,"ondisk_log_size":2460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20133024,"num_objects":2458,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2458,"num_write_kb":19662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2541","reported_seq":2595,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.125153+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:39.125153+0000","last_peered":"2024-09-10T20:20:39.125153+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:39.125153+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20770544,"num_objects":2536,"num_object_clones":0,"num_object_copies":10144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2601","reported_seq":2643,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.309753+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:39.309753+0000","last_peered":"2024-09-10T20:20:39.309753+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:39.309753+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21297744,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2600,"num_write_kb":20799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2492","reported_seq":2566,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.771629+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:36.771629+0000","last_peered":"2024-09-10T20:20:36.771629+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:36.771629+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2492,"log_dups_size":0,"ondisk_log_size":2492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20404816,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":19927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2569","reported_seq":2619,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.245811+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:39.245811+0000","last_peered":"2024-09-10T20:20:39.245811+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:39.245811+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009568,"num_objects":2565,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2540","reported_seq":2617,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.993575+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:36.993575+0000","last_peered":"2024-09-10T20:20:36.993575+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:36.993575+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2540,"log_dups_size":0,"ondisk_log_size":2540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20780192,"num_objects":2537,"num_object_clones":0,"num_object_copies":10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2576","reported_seq":2637,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.123667+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:39.123667+0000","last_peered":"2024-09-10T20:20:39.123667+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:39.123667+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21058720,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":334012608,"num_objects":40778,"num_object_clones":0,"num_object_copies":163112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40805,"num_write_kb":326322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":538345472,"data_stored":538345472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40832,"ondisk_log_size":40832,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299332,"kb_used_data":163508,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94072508,"statfs":{"total":96636764160,"available":96330248192,"internally_reserved":0,"allocated":167432192,"data_stored":167256211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1,14,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296044,"kb_used_data":160732,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94075796,"statfs":{"total":96636764160,"available":96333615104,"internally_reserved":0,"allocated":164589568,"data_stored":164413587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87044,"kb_used_data":37300,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94284796,"statfs":{"total":96636764160,"available":96547631104,"internally_reserved":0,"allocated":38195200,"data_stored":38019219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315180,"kb_used_data":162780,"kb_used_omap":4,"kb_used_meta":152379,"kb_avail":94056660,"statfs":{"total":96636764160,"available":96314019840,"internally_reserved":0,"allocated":166686720,"data_stored":166510739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156036815},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167620608,"data_stored":167620608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38371328,"data_stored":38371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164425728,"data_stored":164425728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167927808,"data_stored":167927808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:40.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:40.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 59 20 41544 41524 5.4977 3.17188 0.0773995 0.0283998 2024-09-10T20:20:41.242 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:41.242 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:41.262 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-10T20:20:39.898113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334012608,"num_objects":40778,"num_object_clones":0,"num_object_copies":163112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40805,"num_write_kb":326322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40832,"ondisk_log_size":40832,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":997600,"kb_used_data":524320,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376489760,"statfs":{"total":386547056640,"available":385525514240,"internally_reserved":0,"allocated":536903680,"data_stored":536199756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":484558390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,11,4,5,22,18,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44040192,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":43008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005789"},"pg_stats":[{"pgid":"1.f","version":"32'2544","reported_seq":2585,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.221678+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:39.221678+0000","last_peered":"2024-09-10T20:20:39.221678+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:39.221678+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2544,"log_dups_size":0,"ondisk_log_size":2544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2544,"num_write_kb":20352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2594","reported_seq":2656,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.100847+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:39.100847+0000","last_peered":"2024-09-10T20:20:39.100847+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:39.100847+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2594,"log_dups_size":0,"ondisk_log_size":2594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214368,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":20732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2488","reported_seq":2563,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.755154+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:36.755154+0000","last_peered":"2024-09-10T20:20:36.755154+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:36.755154+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2488,"log_dups_size":0,"ondisk_log_size":2488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20355664,"num_objects":2485,"num_object_clones":0,"num_object_copies":9940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":19886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2593","reported_seq":2655,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:38.972389+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:38.972389+0000","last_peered":"2024-09-10T20:20:38.972389+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:38.972389+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206176,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2591,"num_write_kb":20717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2610","reported_seq":2661,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.235585+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:39.235585+0000","last_peered":"2024-09-10T20:20:39.235585+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:39.235585+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2610,"log_dups_size":0,"ondisk_log_size":2610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21353632,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2550","reported_seq":2591,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.127912+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:39.127912+0000","last_peered":"2024-09-10T20:20:39.127912+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:39.127912+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20889600,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":20400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2561","reported_seq":2625,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.132213+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:39.132213+0000","last_peered":"2024-09-10T20:20:39.132213+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:39.132213+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20927648,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2577","reported_seq":2640,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.092539+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:39.092539+0000","last_peered":"2024-09-10T20:20:39.092539+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:39.092539+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2577,"log_dups_size":0,"ondisk_log_size":2577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21066912,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2575,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2536","reported_seq":3720,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.300114+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:39.300114+0000","last_peered":"2024-09-10T20:20:39.300114+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:39.300114+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20703552,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":20241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2460","reported_seq":2536,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.969096+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:36.969096+0000","last_peered":"2024-09-10T20:20:36.969096+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:36.969096+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2460,"log_dups_size":0,"ondisk_log_size":2460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20133024,"num_objects":2458,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2458,"num_write_kb":19662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2541","reported_seq":2595,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.125153+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:39.125153+0000","last_peered":"2024-09-10T20:20:39.125153+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:39.125153+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20770544,"num_objects":2536,"num_object_clones":0,"num_object_copies":10144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2601","reported_seq":2643,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.309753+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:39.309753+0000","last_peered":"2024-09-10T20:20:39.309753+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:39.309753+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21297744,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2600,"num_write_kb":20799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2492","reported_seq":2566,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.771629+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:36.771629+0000","last_peered":"2024-09-10T20:20:36.771629+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:36.771629+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2492,"log_dups_size":0,"ondisk_log_size":2492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20404816,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":19927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2569","reported_seq":2619,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.245811+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:39.245811+0000","last_peered":"2024-09-10T20:20:39.245811+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:39.245811+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009568,"num_objects":2565,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2540","reported_seq":2617,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.993575+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:36.993575+0000","last_peered":"2024-09-10T20:20:36.993575+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:36.993575+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2540,"log_dups_size":0,"ondisk_log_size":2540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20780192,"num_objects":2537,"num_object_clones":0,"num_object_copies":10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2576","reported_seq":2637,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.123667+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:39.123667+0000","last_peered":"2024-09-10T20:20:39.123667+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:39.123667+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21058720,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":334012608,"num_objects":40778,"num_object_clones":0,"num_object_copies":163112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40805,"num_write_kb":326322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":538345472,"data_stored":538345472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40832,"ondisk_log_size":40832,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299332,"kb_used_data":163508,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94072508,"statfs":{"total":96636764160,"available":96330248192,"internally_reserved":0,"allocated":167432192,"data_stored":167256211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1,14,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296044,"kb_used_data":160732,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94075796,"statfs":{"total":96636764160,"available":96333615104,"internally_reserved":0,"allocated":164589568,"data_stored":164413587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87044,"kb_used_data":37300,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94284796,"statfs":{"total":96636764160,"available":96547631104,"internally_reserved":0,"allocated":38195200,"data_stored":38019219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315180,"kb_used_data":162780,"kb_used_omap":4,"kb_used_meta":152379,"kb_avail":94056660,"statfs":{"total":96636764160,"available":96314019840,"internally_reserved":0,"allocated":166686720,"data_stored":166510739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156036815},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167620608,"data_stored":167620608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38371328,"data_stored":38371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164425728,"data_stored":164425728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167927808,"data_stored":167927808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:41.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:41.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2544", 'reported_seq': 2585, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.221678+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:39.221678+0000', 'last_peered': '2024-09-10T20:20:39.221678+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:39.221678+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2544, 'log_dups_size': 0, 'ondisk_log_size': 2544, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20840448, 'num_objects': 2544, 'num_object_clones': 0, 'num_object_copies': 10176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2544, 'num_write_kb': 20352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 578}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:41.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:41.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2594", 'reported_seq': 2656, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.100847+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:39.100847+0000', 'last_peered': '2024-09-10T20:20:39.100847+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:39.100847+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2594, 'log_dups_size': 0, 'ondisk_log_size': 2594, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21214368, 'num_objects': 2590, 'num_object_clones': 0, 'num_object_copies': 10360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2592, 'num_write_kb': 20732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 599}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2488", 'reported_seq': 2563, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:36.755154+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:36.755154+0000', 'last_peered': '2024-09-10T20:20:36.755154+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:36.755154+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2488, 'log_dups_size': 0, 'ondisk_log_size': 2488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20355664, 'num_objects': 2485, 'num_object_clones': 0, 'num_object_copies': 9940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2487, 'num_write_kb': 19886, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 523}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2593", 'reported_seq': 2655, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:38.972389+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:38.972389+0000', 'last_peered': '2024-09-10T20:20:38.972389+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:38.972389+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2593, 'log_dups_size': 0, 'ondisk_log_size': 2593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21206176, 'num_objects': 2589, 'num_object_clones': 0, 'num_object_copies': 10356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2591, 'num_write_kb': 20717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2610", 'reported_seq': 2661, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.235585+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:39.235585+0000', 'last_peered': '2024-09-10T20:20:39.235585+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:39.235585+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2610, 'log_dups_size': 0, 'ondisk_log_size': 2610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21353632, 'num_objects': 2607, 'num_object_clones': 0, 'num_object_copies': 10428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2608, 'num_write_kb': 20861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 627}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:41.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2550", 'reported_seq': 2591, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.127912+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:39.127912+0000', 'last_peered': '2024-09-10T20:20:39.127912+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:39.127912+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2550, 'log_dups_size': 0, 'ondisk_log_size': 2550, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20889600, 'num_objects': 2550, 'num_object_clones': 0, 'num_object_copies': 10200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2550, 'num_write_kb': 20400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 612}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2561", 'reported_seq': 2625, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.132213+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:39.132213+0000', 'last_peered': '2024-09-10T20:20:39.132213+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:39.132213+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2561, 'log_dups_size': 0, 'ondisk_log_size': 2561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20927648, 'num_objects': 2555, 'num_object_clones': 0, 'num_object_copies': 10220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2559, 'num_write_kb': 20452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 602}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2577", 'reported_seq': 2640, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.092539+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:39.092539+0000', 'last_peered': '2024-09-10T20:20:39.092539+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:39.092539+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2577, 'log_dups_size': 0, 'ondisk_log_size': 2577, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21066912, 'num_objects': 2572, 'num_object_clones': 0, 'num_object_copies': 10288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2575, 'num_write_kb': 20588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 529}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2536", 'reported_seq': 3720, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.300114+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:39.300114+0000', 'last_peered': '2024-09-10T20:20:39.300114+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:39.300114+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2536, 'log_dups_size': 0, 'ondisk_log_size': 2536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20703552, 'num_objects': 2528, 'num_object_clones': 0, 'num_object_copies': 10112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1355, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2533, 'num_write_kb': 20241, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 568, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1173}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:41.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2460", 'reported_seq': 2536, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:36.969096+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:36.969096+0000', 'last_peered': '2024-09-10T20:20:36.969096+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:36.969096+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2460, 'log_dups_size': 0, 'ondisk_log_size': 2460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20133024, 'num_objects': 2458, 'num_object_clones': 0, 'num_object_copies': 9832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2458, 'num_write_kb': 19662, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 555}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2541", 'reported_seq': 2595, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.125153+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:39.125153+0000', 'last_peered': '2024-09-10T20:20:39.125153+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:39.125153+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2541, 'log_dups_size': 0, 'ondisk_log_size': 2541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20770544, 'num_objects': 2536, 'num_object_clones': 0, 'num_object_copies': 10144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2538, 'num_write_kb': 20292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 580}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2601", 'reported_seq': 2643, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.309753+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:39.309753+0000', 'last_peered': '2024-09-10T20:20:39.309753+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:39.309753+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2601, 'log_dups_size': 0, 'ondisk_log_size': 2601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21297744, 'num_objects': 2600, 'num_object_clones': 0, 'num_object_copies': 10400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2600, 'num_write_kb': 20799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2492", 'reported_seq': 2566, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:36.771629+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:36.771629+0000', 'last_peered': '2024-09-10T20:20:36.771629+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:36.771629+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2492, 'log_dups_size': 0, 'ondisk_log_size': 2492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20404816, 'num_objects': 2491, 'num_object_clones': 0, 'num_object_copies': 9964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2491, 'num_write_kb': 19927, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 543}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2569", 'reported_seq': 2619, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.245811+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:39.245811+0000', 'last_peered': '2024-09-10T20:20:39.245811+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:39.245811+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2569, 'log_dups_size': 0, 'ondisk_log_size': 2569, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21009568, 'num_objects': 2565, 'num_object_clones': 0, 'num_object_copies': 10260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2567, 'num_write_kb': 20532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:41.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2540", 'reported_seq': 2617, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:36.993575+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:36.993575+0000', 'last_peered': '2024-09-10T20:20:36.993575+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:36.993575+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2540, 'log_dups_size': 0, 'ondisk_log_size': 2540, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20780192, 'num_objects': 2537, 'num_object_clones': 0, 'num_object_copies': 10148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2538, 'num_write_kb': 20301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:41.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:41.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2576", 'reported_seq': 2637, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:39.123667+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:39.123667+0000', 'last_peered': '2024-09-10T20:20:39.123667+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:39.123667+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2576, 'log_dups_size': 0, 'ondisk_log_size': 2576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21058720, 'num_objects': 2571, 'num_object_clones': 0, 'num_object_copies': 10284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2574, 'num_write_kb': 20580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:41.267 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:41.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:20:41.774636+0000 min lat: 0.00230373 max lat: 2.76361 avg lat: 0.0286491 2024-09-10T20:20:41.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:20:41.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 60 20 41888 41868 5.45086 2.6875 0.0337486 0.0286491 2024-09-10T20:20:41.945 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:41.945 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30916/163112 objects degraded (18.954%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":130,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40778,"data_bytes":334012608,"bytes_used":1021542400,"bytes_avail":385525514240,"bytes_total":386547056640,"degraded_objects":30916,"degraded_total":163112,"degraded_ratio":0.18953847662955514,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":312510,"recovering_keys_per_sec":0,"num_objects_recovered":458,"num_bytes_recovered":3751936,"num_keys_recovered":0,"write_bytes_sec":3668246,"read_op_per_sec":0,"write_op_per_sec":447},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:41.976 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:41.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:42.529 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:42.530 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:42.559 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-10T20:20:41.898590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334012608,"num_objects":40778,"num_object_clones":0,"num_object_copies":163112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40805,"num_write_kb":326322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40832,"ondisk_log_size":40832,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":997600,"kb_used_data":524320,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376489760,"statfs":{"total":386547056640,"available":385525514240,"internally_reserved":0,"allocated":536903680,"data_stored":536199756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":484558390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,11,4,5,22,18,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34029568,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4154,"num_write_kb":33232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005303"},"pg_stats":[{"pgid":"1.f","version":"32'2544","reported_seq":2585,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.221678+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:39.221678+0000","last_peered":"2024-09-10T20:20:39.221678+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:39.221678+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2544,"log_dups_size":0,"ondisk_log_size":2544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2544,"num_write_kb":20352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2594","reported_seq":2656,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.100847+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:39.100847+0000","last_peered":"2024-09-10T20:20:39.100847+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:39.100847+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2594,"log_dups_size":0,"ondisk_log_size":2594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214368,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":20732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2488","reported_seq":2563,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.755154+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:36.755154+0000","last_peered":"2024-09-10T20:20:36.755154+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:36.755154+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2488,"log_dups_size":0,"ondisk_log_size":2488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20355664,"num_objects":2485,"num_object_clones":0,"num_object_copies":9940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":19886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2593","reported_seq":2655,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:38.972389+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:38.972389+0000","last_peered":"2024-09-10T20:20:38.972389+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:38.972389+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206176,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2591,"num_write_kb":20717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2610","reported_seq":2661,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.235585+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:39.235585+0000","last_peered":"2024-09-10T20:20:39.235585+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:39.235585+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2610,"log_dups_size":0,"ondisk_log_size":2610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21353632,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2550","reported_seq":2591,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.127912+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:39.127912+0000","last_peered":"2024-09-10T20:20:39.127912+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:39.127912+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20889600,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":20400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2561","reported_seq":2625,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.132213+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:39.132213+0000","last_peered":"2024-09-10T20:20:39.132213+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:39.132213+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20927648,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2577","reported_seq":2640,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.092539+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:39.092539+0000","last_peered":"2024-09-10T20:20:39.092539+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:39.092539+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2577,"log_dups_size":0,"ondisk_log_size":2577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21066912,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2575,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2536","reported_seq":3720,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.300114+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:39.300114+0000","last_peered":"2024-09-10T20:20:39.300114+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:39.300114+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20703552,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":20241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2460","reported_seq":2536,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.969096+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:36.969096+0000","last_peered":"2024-09-10T20:20:36.969096+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:36.969096+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2460,"log_dups_size":0,"ondisk_log_size":2460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20133024,"num_objects":2458,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2458,"num_write_kb":19662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2541","reported_seq":2595,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.125153+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:39.125153+0000","last_peered":"2024-09-10T20:20:39.125153+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:39.125153+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20770544,"num_objects":2536,"num_object_clones":0,"num_object_copies":10144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2601","reported_seq":2643,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.309753+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:39.309753+0000","last_peered":"2024-09-10T20:20:39.309753+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:39.309753+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21297744,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2600,"num_write_kb":20799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2492","reported_seq":2566,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.771629+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:36.771629+0000","last_peered":"2024-09-10T20:20:36.771629+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:36.771629+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2492,"log_dups_size":0,"ondisk_log_size":2492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20404816,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":19927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2569","reported_seq":2619,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.245811+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:39.245811+0000","last_peered":"2024-09-10T20:20:39.245811+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:39.245811+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009568,"num_objects":2565,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2540","reported_seq":2617,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:36.993575+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:36.993575+0000","last_peered":"2024-09-10T20:20:36.993575+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:36.993575+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2540,"log_dups_size":0,"ondisk_log_size":2540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20780192,"num_objects":2537,"num_object_clones":0,"num_object_copies":10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2576","reported_seq":2637,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:39.123667+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:39.123667+0000","last_peered":"2024-09-10T20:20:39.123667+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:39.123667+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21058720,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":334012608,"num_objects":40778,"num_object_clones":0,"num_object_copies":163112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40805,"num_write_kb":326322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":538345472,"data_stored":538345472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40832,"ondisk_log_size":40832,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299332,"kb_used_data":163508,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94072508,"statfs":{"total":96636764160,"available":96330248192,"internally_reserved":0,"allocated":167432192,"data_stored":167256211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1,14,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296044,"kb_used_data":160732,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94075796,"statfs":{"total":96636764160,"available":96333615104,"internally_reserved":0,"allocated":164589568,"data_stored":164413587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87044,"kb_used_data":37300,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94284796,"statfs":{"total":96636764160,"available":96547631104,"internally_reserved":0,"allocated":38195200,"data_stored":38019219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315180,"kb_used_data":162780,"kb_used_omap":4,"kb_used_meta":152379,"kb_avail":94056660,"statfs":{"total":96636764160,"available":96314019840,"internally_reserved":0,"allocated":166686720,"data_stored":166510739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156036815},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167620608,"data_stored":167620608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38371328,"data_stored":38371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164425728,"data_stored":164425728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167927808,"data_stored":167927808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:42.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 61 20 42298 42278 5.41401 3.20312 0.0369661 0.0288431 2024-09-10T20:20:43.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 62 20 42758 42738 5.38464 3.59375 0.0486341 0.0290014 2024-09-10T20:20:44.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 63 20 43209 43189 5.3551 3.52344 0.0292391 0.0291603 2024-09-10T20:20:45.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:45.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 64 20 43652 43632 5.3255 3.46094 0.0389184 0.0293284 2024-09-10T20:20:46.116 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:46.116 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-10T20:20:45.899650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350191808,"num_objects":42753,"num_object_clones":0,"num_object_copies":171012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42780,"num_write_kb":342122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42807,"ondisk_log_size":42807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1064784,"kb_used_data":555408,"kb_used_omap":14,"kb_used_meta":509297,"kb_avail":376422576,"statfs":{"total":386547056640,"available":385456717824,"internally_reserved":0,"allocated":568737792,"data_stored":568033868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":521520694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,5,2,3,1,11,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45498368,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":2457600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005435"},"pg_stats":[{"pgid":"1.f","version":"32'2661","reported_seq":2702,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.258810+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:44.258810+0000","last_peered":"2024-09-10T20:20:44.258810+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:44.258810+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2661,"log_dups_size":0,"ondisk_log_size":2661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21798912,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":21288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2737","reported_seq":2799,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.127908+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:44.127908+0000","last_peered":"2024-09-10T20:20:44.127908+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:44.127908+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22385824,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2586","reported_seq":2661,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.865967+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:41.865967+0000","last_peered":"2024-09-10T20:20:41.865967+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:41.865967+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2586,"log_dups_size":0,"ondisk_log_size":2586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21158480,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":20670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":621}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2736","reported_seq":2798,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.152537+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:44.152537+0000","last_peered":"2024-09-10T20:20:44.152537+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:44.152537+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22377632,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2733","reported_seq":2782,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.334619+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:44.334619+0000","last_peered":"2024-09-10T20:20:44.334619+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:44.334619+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22361248,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":21845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2688","reported_seq":2729,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.329793+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:44.329793+0000","last_peered":"2024-09-10T20:20:44.329793+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:44.329793+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":21504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2671","reported_seq":2735,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.068867+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:44.068867+0000","last_peered":"2024-09-10T20:20:44.068867+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:44.068867+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828768,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":712}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2724","reported_seq":2787,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.158235+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:44.158235+0000","last_peered":"2024-09-10T20:20:44.158235+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:44.158235+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22271136,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2722,"num_write_kb":21764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2672","reported_seq":4038,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.359299+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:44.359299+0000","last_peered":"2024-09-10T20:20:44.359299+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:44.359299+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2672,"log_dups_size":0,"ondisk_log_size":2672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21817664,"num_objects":2664,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":658,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2559","reported_seq":2635,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.954556+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:41.954556+0000","last_peered":"2024-09-10T20:20:41.954556+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:41.954556+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":654}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2662","reported_seq":2716,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.301661+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:44.301661+0000","last_peered":"2024-09-10T20:20:44.301661+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:44.301661+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2662,"log_dups_size":0,"ondisk_log_size":2662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21761776,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2659,"num_write_kb":21260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2720","reported_seq":2763,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.316442+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:44.316442+0000","last_peered":"2024-09-10T20:20:44.316442+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:44.316442+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2720,"log_dups_size":0,"ondisk_log_size":2720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22272592,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":741}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2598","reported_seq":2670,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.940355+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:41.940355+0000","last_peered":"2024-09-10T20:20:41.940355+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:41.940355+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2598,"log_dups_size":0,"ondisk_log_size":2598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21273168,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":20775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2695","reported_seq":2745,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.336324+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:44.336324+0000","last_peered":"2024-09-10T20:20:44.336324+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:44.336324+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041760,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2664","reported_seq":2742,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.896940+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:41.896940+0000","last_peered":"2024-09-10T20:20:41.896940+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:41.896940+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":21293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2701","reported_seq":2764,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.112961+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:44.112961+0000","last_peered":"2024-09-10T20:20:44.112961+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:44.112961+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2701,"log_dups_size":0,"ondisk_log_size":2701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082720,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":21580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":350191808,"num_objects":42753,"num_object_clones":0,"num_object_copies":171012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42780,"num_write_kb":342122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":570699776,"data_stored":570699776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42807,"ondisk_log_size":42807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311788,"kb_used_data":171804,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94060052,"statfs":{"total":96636764160,"available":96317493248,"internally_reserved":0,"allocated":175927296,"data_stored":175751315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,5,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307400,"kb_used_data":167928,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94064440,"statfs":{"total":96636764160,"available":96321986560,"internally_reserved":0,"allocated":171958272,"data_stored":171782291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135372,"kb_used_data":45372,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94236468,"statfs":{"total":96636764160,"available":96498143232,"internally_reserved":0,"allocated":46460928,"data_stored":46284947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":92139218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310224,"kb_used_data":170304,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94061616,"statfs":{"total":96636764160,"available":96319094784,"internally_reserved":0,"allocated":174391296,"data_stored":174215315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175898624,"data_stored":175898624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46641152,"data_stored":46641152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171937792,"data_stored":171937792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176222208,"data_stored":176222208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:46.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:46.723 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:46.723 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:46.744 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-10T20:20:45.899650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350191808,"num_objects":42753,"num_object_clones":0,"num_object_copies":171012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42780,"num_write_kb":342122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42807,"ondisk_log_size":42807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1064784,"kb_used_data":555408,"kb_used_omap":14,"kb_used_meta":509297,"kb_avail":376422576,"statfs":{"total":386547056640,"available":385456717824,"internally_reserved":0,"allocated":568737792,"data_stored":568033868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":521520694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,5,2,3,1,11,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45498368,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":2457600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005435"},"pg_stats":[{"pgid":"1.f","version":"32'2661","reported_seq":2702,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.258810+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:44.258810+0000","last_peered":"2024-09-10T20:20:44.258810+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:44.258810+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2661,"log_dups_size":0,"ondisk_log_size":2661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21798912,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":21288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2737","reported_seq":2799,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.127908+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:44.127908+0000","last_peered":"2024-09-10T20:20:44.127908+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:44.127908+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22385824,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2586","reported_seq":2661,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.865967+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:41.865967+0000","last_peered":"2024-09-10T20:20:41.865967+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:41.865967+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2586,"log_dups_size":0,"ondisk_log_size":2586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21158480,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":20670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":621}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2736","reported_seq":2798,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.152537+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:44.152537+0000","last_peered":"2024-09-10T20:20:44.152537+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:44.152537+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22377632,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2733","reported_seq":2782,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.334619+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:44.334619+0000","last_peered":"2024-09-10T20:20:44.334619+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:44.334619+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22361248,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":21845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2688","reported_seq":2729,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.329793+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:44.329793+0000","last_peered":"2024-09-10T20:20:44.329793+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:44.329793+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":21504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2671","reported_seq":2735,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.068867+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:44.068867+0000","last_peered":"2024-09-10T20:20:44.068867+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:44.068867+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828768,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":712}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2724","reported_seq":2787,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.158235+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:44.158235+0000","last_peered":"2024-09-10T20:20:44.158235+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:44.158235+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22271136,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2722,"num_write_kb":21764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2672","reported_seq":4038,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.359299+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:44.359299+0000","last_peered":"2024-09-10T20:20:44.359299+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:44.359299+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2672,"log_dups_size":0,"ondisk_log_size":2672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21817664,"num_objects":2664,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":658,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2559","reported_seq":2635,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.954556+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:41.954556+0000","last_peered":"2024-09-10T20:20:41.954556+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:41.954556+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":654}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2662","reported_seq":2716,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.301661+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:44.301661+0000","last_peered":"2024-09-10T20:20:44.301661+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:44.301661+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2662,"log_dups_size":0,"ondisk_log_size":2662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21761776,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2659,"num_write_kb":21260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2720","reported_seq":2763,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.316442+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:44.316442+0000","last_peered":"2024-09-10T20:20:44.316442+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:44.316442+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2720,"log_dups_size":0,"ondisk_log_size":2720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22272592,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":741}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2598","reported_seq":2670,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.940355+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:41.940355+0000","last_peered":"2024-09-10T20:20:41.940355+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:41.940355+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2598,"log_dups_size":0,"ondisk_log_size":2598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21273168,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":20775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2695","reported_seq":2745,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.336324+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:44.336324+0000","last_peered":"2024-09-10T20:20:44.336324+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:44.336324+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041760,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2664","reported_seq":2742,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:41.896940+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:41.896940+0000","last_peered":"2024-09-10T20:20:41.896940+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:41.896940+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":21293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2701","reported_seq":2764,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.112961+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:44.112961+0000","last_peered":"2024-09-10T20:20:44.112961+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:44.112961+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2701,"log_dups_size":0,"ondisk_log_size":2701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082720,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":21580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":350191808,"num_objects":42753,"num_object_clones":0,"num_object_copies":171012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42780,"num_write_kb":342122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":570699776,"data_stored":570699776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42807,"ondisk_log_size":42807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311788,"kb_used_data":171804,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94060052,"statfs":{"total":96636764160,"available":96317493248,"internally_reserved":0,"allocated":175927296,"data_stored":175751315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,5,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307400,"kb_used_data":167928,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94064440,"statfs":{"total":96636764160,"available":96321986560,"internally_reserved":0,"allocated":171958272,"data_stored":171782291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135372,"kb_used_data":45372,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94236468,"statfs":{"total":96636764160,"available":96498143232,"internally_reserved":0,"allocated":46460928,"data_stored":46284947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":92139218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310224,"kb_used_data":170304,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94061616,"statfs":{"total":96636764160,"available":96319094784,"internally_reserved":0,"allocated":174391296,"data_stored":174215315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175898624,"data_stored":175898624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46641152,"data_stored":46641152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171937792,"data_stored":171937792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176222208,"data_stored":176222208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:46.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:46.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2661", 'reported_seq': 2702, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.258810+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:44.258810+0000', 'last_peered': '2024-09-10T20:20:44.258810+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:44.258810+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2661, 'log_dups_size': 0, 'ondisk_log_size': 2661, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21798912, 'num_objects': 2661, 'num_object_clones': 0, 'num_object_copies': 10644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2661, 'num_write_kb': 21288, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 695}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:46.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:46.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2737", 'reported_seq': 2799, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.127908+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:44.127908+0000', 'last_peered': '2024-09-10T20:20:44.127908+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:44.127908+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2737, 'log_dups_size': 0, 'ondisk_log_size': 2737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22385824, 'num_objects': 2733, 'num_object_clones': 0, 'num_object_copies': 10932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2735, 'num_write_kb': 21876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 742}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2586", 'reported_seq': 2661, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:41.865967+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:41.865967+0000', 'last_peered': '2024-09-10T20:20:41.865967+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:41.865967+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2586, 'log_dups_size': 0, 'ondisk_log_size': 2586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21158480, 'num_objects': 2583, 'num_object_clones': 0, 'num_object_copies': 10332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2585, 'num_write_kb': 20670, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 621}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2736", 'reported_seq': 2798, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.152537+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:44.152537+0000', 'last_peered': '2024-09-10T20:20:44.152537+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:44.152537+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2736, 'log_dups_size': 0, 'ondisk_log_size': 2736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22377632, 'num_objects': 2732, 'num_object_clones': 0, 'num_object_copies': 10928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2734, 'num_write_kb': 21861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 729}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2733", 'reported_seq': 2782, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.334619+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:44.334619+0000', 'last_peered': '2024-09-10T20:20:44.334619+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:44.334619+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2733, 'log_dups_size': 0, 'ondisk_log_size': 2733, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22361248, 'num_objects': 2730, 'num_object_clones': 0, 'num_object_copies': 10920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2731, 'num_write_kb': 21845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 750}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:46.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2688", 'reported_seq': 2729, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.329793+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:44.329793+0000', 'last_peered': '2024-09-10T20:20:44.329793+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:44.329793+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2688, 'log_dups_size': 0, 'ondisk_log_size': 2688, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22020096, 'num_objects': 2688, 'num_object_clones': 0, 'num_object_copies': 10752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2688, 'num_write_kb': 21504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 750}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2671", 'reported_seq': 2735, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.068867+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:44.068867+0000', 'last_peered': '2024-09-10T20:20:44.068867+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:44.068867+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2671, 'log_dups_size': 0, 'ondisk_log_size': 2671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21828768, 'num_objects': 2665, 'num_object_clones': 0, 'num_object_copies': 10660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2669, 'num_write_kb': 21332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 712}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2724", 'reported_seq': 2787, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.158235+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:44.158235+0000', 'last_peered': '2024-09-10T20:20:44.158235+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:44.158235+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2724, 'log_dups_size': 0, 'ondisk_log_size': 2724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22271136, 'num_objects': 2719, 'num_object_clones': 0, 'num_object_copies': 10876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2722, 'num_write_kb': 21764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2672", 'reported_seq': 4038, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.359299+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:44.359299+0000', 'last_peered': '2024-09-10T20:20:44.359299+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:44.359299+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2672, 'log_dups_size': 0, 'ondisk_log_size': 2672, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21817664, 'num_objects': 2664, 'num_object_clones': 0, 'num_object_copies': 10656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2669, 'num_write_kb': 21329, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 658, 'num_bytes_recovered': 5365760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1399}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2559", 'reported_seq': 2635, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:41.954556+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:41.954556+0000', 'last_peered': '2024-09-10T20:20:41.954556+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:41.954556+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2559, 'log_dups_size': 0, 'ondisk_log_size': 2559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20944032, 'num_objects': 2557, 'num_object_clones': 0, 'num_object_copies': 10228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2557, 'num_write_kb': 20454, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 654}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:46.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2662", 'reported_seq': 2716, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.301661+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:44.301661+0000', 'last_peered': '2024-09-10T20:20:44.301661+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:44.301661+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2662, 'log_dups_size': 0, 'ondisk_log_size': 2662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21761776, 'num_objects': 2657, 'num_object_clones': 0, 'num_object_copies': 10628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2659, 'num_write_kb': 21260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 701}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2720", 'reported_seq': 2763, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.316442+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:44.316442+0000', 'last_peered': '2024-09-10T20:20:44.316442+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:44.316442+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2720, 'log_dups_size': 0, 'ondisk_log_size': 2720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22272592, 'num_objects': 2719, 'num_object_clones': 0, 'num_object_copies': 10876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2719, 'num_write_kb': 21751, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 741}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2598", 'reported_seq': 2670, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:41.940355+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:41.940355+0000', 'last_peered': '2024-09-10T20:20:41.940355+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:41.940355+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2598, 'log_dups_size': 0, 'ondisk_log_size': 2598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21273168, 'num_objects': 2597, 'num_object_clones': 0, 'num_object_copies': 10388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2597, 'num_write_kb': 20775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2695", 'reported_seq': 2745, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.336324+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:44.336324+0000', 'last_peered': '2024-09-10T20:20:44.336324+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:44.336324+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2695, 'log_dups_size': 0, 'ondisk_log_size': 2695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22041760, 'num_objects': 2691, 'num_object_clones': 0, 'num_object_copies': 10764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2693, 'num_write_kb': 21540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 716}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2664", 'reported_seq': 2742, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:41.896940+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:41.896940+0000', 'last_peered': '2024-09-10T20:20:41.896940+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:41.896940+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2664, 'log_dups_size': 0, 'ondisk_log_size': 2664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21796000, 'num_objects': 2661, 'num_object_clones': 0, 'num_object_copies': 10644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2662, 'num_write_kb': 21293, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2701", 'reported_seq': 2764, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:44.112961+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:44.112961+0000', 'last_peered': '2024-09-10T20:20:44.112961+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:44.112961+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2701, 'log_dups_size': 0, 'ondisk_log_size': 2701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22082720, 'num_objects': 2696, 'num_object_clones': 0, 'num_object_copies': 10784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2699, 'num_write_kb': 21580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:46.749 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:46.777 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 65 20 44033 44013 5.28936 2.97656 0.0311566 0.0295222 2024-09-10T20:20:47.476 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:47.476 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30826/171012 objects degraded (18.026%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":136,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42753,"data_bytes":350191808,"bytes_used":1090338816,"bytes_avail":385456717824,"bytes_total":386547056640,"degraded_objects":30826,"degraded_total":171012,"degraded_ratio":0.18025635627909153,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":204707,"recovering_keys_per_sec":0,"num_objects_recovered":300,"num_bytes_recovered":2457600,"num_keys_recovered":0,"write_bytes_sec":3789814,"read_op_per_sec":0,"write_op_per_sec":462},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:47.506 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:47.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:47.778 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 66 20 44421 44401 5.25514 3.03125 0.0741015 0.0297181 2024-09-10T20:20:48.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:48.109 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:48.134 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-10T20:20:47.900108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":354640064,"num_objects":43296,"num_object_clones":0,"num_object_copies":173184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43323,"num_write_kb":346466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43350,"ondisk_log_size":43350,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1090552,"kb_used_data":572856,"kb_used_omap":14,"kb_used_meta":517617,"kb_avail":376396808,"statfs":{"total":386547056640,"available":385430331392,"internally_reserved":0,"allocated":586604544,"data_stored":585900620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":530040374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,5,19,12,16],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36519936,"num_objects":4458,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4458,"num_write_kb":35664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1884160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002954"},"pg_stats":[{"pgid":"1.f","version":"32'2661","reported_seq":2702,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.258810+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:44.258810+0000","last_peered":"2024-09-10T20:20:44.258810+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:44.258810+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2661,"log_dups_size":0,"ondisk_log_size":2661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21798912,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":21288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2737","reported_seq":2799,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.127908+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:44.127908+0000","last_peered":"2024-09-10T20:20:44.127908+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:44.127908+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22385824,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2736","reported_seq":2811,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.960318+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:46.960318+0000","last_peered":"2024-09-10T20:20:46.960318+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:46.960318+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22387280,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":771}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2736","reported_seq":2798,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.152537+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:44.152537+0000","last_peered":"2024-09-10T20:20:44.152537+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:44.152537+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22377632,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2733","reported_seq":2782,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.334619+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:44.334619+0000","last_peered":"2024-09-10T20:20:44.334619+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:44.334619+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22361248,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":21845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2688","reported_seq":2729,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.329793+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:44.329793+0000","last_peered":"2024-09-10T20:20:44.329793+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:44.329793+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":21504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2671","reported_seq":2735,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.068867+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:44.068867+0000","last_peered":"2024-09-10T20:20:44.068867+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:44.068867+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828768,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":712}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2724","reported_seq":2787,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.158235+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:44.158235+0000","last_peered":"2024-09-10T20:20:44.158235+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:44.158235+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22271136,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2722,"num_write_kb":21764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2672","reported_seq":4038,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.359299+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:44.359299+0000","last_peered":"2024-09-10T20:20:44.359299+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:44.359299+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2672,"log_dups_size":0,"ondisk_log_size":2672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21817664,"num_objects":2664,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":658,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2690","reported_seq":2766,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.897414+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:46.897414+0000","last_peered":"2024-09-10T20:20:46.897414+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:46.897414+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2690,"log_dups_size":0,"ondisk_log_size":2690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22017184,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":21502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2662","reported_seq":2716,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.301661+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:44.301661+0000","last_peered":"2024-09-10T20:20:44.301661+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:44.301661+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2662,"log_dups_size":0,"ondisk_log_size":2662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21761776,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2659,"num_write_kb":21260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2720","reported_seq":2763,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.316442+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:44.316442+0000","last_peered":"2024-09-10T20:20:44.316442+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:44.316442+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2720,"log_dups_size":0,"ondisk_log_size":2720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22272592,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":741}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2722","reported_seq":2796,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.957497+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:46.957497+0000","last_peered":"2024-09-10T20:20:46.957497+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:46.957497+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22288976,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":21767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2695","reported_seq":2745,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.336324+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:44.336324+0000","last_peered":"2024-09-10T20:20:44.336324+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:44.336324+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041760,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2802","reported_seq":2880,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.872333+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:46.872333+0000","last_peered":"2024-09-10T20:20:46.872333+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:46.872333+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22926496,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":22397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2701","reported_seq":2764,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:44.112961+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:44.112961+0000","last_peered":"2024-09-10T20:20:44.112961+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:44.112961+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2701,"log_dups_size":0,"ondisk_log_size":2701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082720,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":21580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":354640064,"num_objects":43296,"num_object_clones":0,"num_object_copies":173184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43323,"num_write_kb":346466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5365760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":588578816,"data_stored":588578816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43350,"ondisk_log_size":43350,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311788,"kb_used_data":171804,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94060052,"statfs":{"total":96636764160,"available":96317493248,"internally_reserved":0,"allocated":175927296,"data_stored":175751315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,5,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320004,"kb_used_data":176372,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94051836,"statfs":{"total":96636764160,"available":96309080064,"internally_reserved":0,"allocated":180604928,"data_stored":180428947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148536,"kb_used_data":54376,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94223304,"statfs":{"total":96636764160,"available":96484663296,"internally_reserved":0,"allocated":55681024,"data_stored":55505043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310224,"kb_used_data":170304,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94061616,"statfs":{"total":96636764160,"available":96319094784,"internally_reserved":0,"allocated":174391296,"data_stored":174215315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175898624,"data_stored":175898624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55705600,"data_stored":55705600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180752384,"data_stored":180752384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176222208,"data_stored":176222208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:48.778 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 67 20 44844 44824 5.22597 3.30469 0.0239062 0.0298836 2024-09-10T20:20:49.778 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 68 20 45286 45266 5.1999 3.45312 0.0475592 0.0300327 2024-09-10T20:20:50.780 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 69 20 45657 45637 5.16643 2.89844 0.0421769 0.0302294 2024-09-10T20:20:51.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:51.724 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:51.725 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-10T20:20:49.900532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367485120,"num_objects":44864,"num_object_clones":0,"num_object_copies":179456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44891,"num_write_kb":359010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44918,"ondisk_log_size":44918,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1115432,"kb_used_data":589412,"kb_used_omap":14,"kb_used_meta":525937,"kb_avail":376371928,"statfs":{"total":386547056640,"available":385404854272,"internally_reserved":0,"allocated":603557888,"data_stored":602853964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":538560054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,12,28,7,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45056000,"num_objects":5500,"num_object_clones":0,"num_object_copies":22000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5500,"num_write_kb":44000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":2473984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002882"},"pg_stats":[{"pgid":"1.f","version":"32'2806","reported_seq":2846,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.365252+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:49.365252+0000","last_peered":"2024-09-10T20:20:49.365252+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:49.365252+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2806,"log_dups_size":0,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22986752,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":22448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2853","reported_seq":2916,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.179992+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:49.179992+0000","last_peered":"2024-09-10T20:20:49.179992+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:49.179992+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336096,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2736","reported_seq":2811,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.960318+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:46.960318+0000","last_peered":"2024-09-10T20:20:46.960318+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:46.960318+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22387280,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":771}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2867","reported_seq":2929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.164509+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:49.164509+0000","last_peered":"2024-09-10T20:20:49.164509+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:49.164509+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23450784,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2846","reported_seq":2896,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.342665+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:49.342665+0000","last_peered":"2024-09-10T20:20:49.342665+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:49.342665+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2846,"log_dups_size":0,"ondisk_log_size":2846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23286944,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2844,"num_write_kb":22749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2840","reported_seq":2880,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.369565+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:49.369565+0000","last_peered":"2024-09-10T20:20:49.369565+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:49.369565+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2840,"num_write_kb":22720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2812","reported_seq":2873,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.195195+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:49.195195+0000","last_peered":"2024-09-10T20:20:49.195195+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:49.195195+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2812,"log_dups_size":0,"ondisk_log_size":2812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22983840,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2810,"num_write_kb":22460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":853}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2839","reported_seq":2902,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.173088+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:49.173088+0000","last_peered":"2024-09-10T20:20:49.173088+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:49.173088+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2839,"log_dups_size":0,"ondisk_log_size":2839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213216,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":22684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2788","reported_seq":4298,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.368105+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:49.368105+0000","last_peered":"2024-09-10T20:20:49.368105+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:49.368105+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22767936,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2785,"num_write_kb":22257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1587}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2690","reported_seq":2766,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.897414+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:46.897414+0000","last_peered":"2024-09-10T20:20:46.897414+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:46.897414+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2690,"log_dups_size":0,"ondisk_log_size":2690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22017184,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":21502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2789","reported_seq":2843,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.272884+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:49.272884+0000","last_peered":"2024-09-10T20:20:49.272884+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:49.272884+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2789,"log_dups_size":0,"ondisk_log_size":2789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22802160,"num_objects":2784,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2854","reported_seq":2897,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.291913+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:49.291913+0000","last_peered":"2024-09-10T20:20:49.291913+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:49.291913+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23370320,"num_objects":2853,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":22823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2722","reported_seq":2796,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.957497+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:46.957497+0000","last_peered":"2024-09-10T20:20:46.957497+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:46.957497+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22288976,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":21767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2840","reported_seq":2890,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.130378+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:49.130378+0000","last_peered":"2024-09-10T20:20:49.130378+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:49.130378+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23229600,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2838,"num_write_kb":22700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":861}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2802","reported_seq":2880,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.872333+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:46.872333+0000","last_peered":"2024-09-10T20:20:46.872333+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:46.872333+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22926496,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":22397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2834","reported_seq":2897,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.156210+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:49.156210+0000","last_peered":"2024-09-10T20:20:49.156210+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:49.156210+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2834,"log_dups_size":0,"ondisk_log_size":2834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172256,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2832,"num_write_kb":22644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367485120,"num_objects":44864,"num_object_clones":0,"num_object_copies":179456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44891,"num_write_kb":359010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":605667328,"data_stored":605667328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44918,"ondisk_log_size":44918,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324140,"kb_used_data":179996,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94047700,"statfs":{"total":96636764160,"available":96304844800,"internally_reserved":0,"allocated":184315904,"data_stored":184139923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320004,"kb_used_data":176372,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94051836,"statfs":{"total":96636764160,"available":96309080064,"internally_reserved":0,"allocated":180604928,"data_stored":180428947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148536,"kb_used_data":54376,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94223304,"statfs":{"total":96636764160,"available":96484663296,"internally_reserved":0,"allocated":55681024,"data_stored":55505043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322752,"kb_used_data":178668,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94049088,"statfs":{"total":96636764160,"available":96306266112,"internally_reserved":0,"allocated":182956032,"data_stored":182780051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,7,4,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184446976,"data_stored":184446976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55705600,"data_stored":55705600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180752384,"data_stored":180752384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184762368,"data_stored":184762368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:51.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:51.780 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 70 20 46062 46042 5.13782 3.16406 0.0391693 0.0303936 2024-09-10T20:20:52.300 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:52.300 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:52.325 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-10T20:20:51.902177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367485120,"num_objects":44864,"num_object_clones":0,"num_object_copies":179456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44891,"num_write_kb":359010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44918,"ondisk_log_size":44918,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1115432,"kb_used_data":589412,"kb_used_omap":14,"kb_used_meta":525937,"kb_avail":376371928,"statfs":{"total":386547056640,"available":385404854272,"internally_reserved":0,"allocated":603557888,"data_stored":602853964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":538560054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,12,28,7,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33472512,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":32688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004064"},"pg_stats":[{"pgid":"1.f","version":"32'2806","reported_seq":2846,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.365252+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:49.365252+0000","last_peered":"2024-09-10T20:20:49.365252+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:49.365252+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2806,"log_dups_size":0,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22986752,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":22448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2853","reported_seq":2916,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.179992+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:49.179992+0000","last_peered":"2024-09-10T20:20:49.179992+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:49.179992+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336096,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2736","reported_seq":2811,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.960318+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:46.960318+0000","last_peered":"2024-09-10T20:20:46.960318+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:46.960318+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22387280,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":771}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2867","reported_seq":2929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.164509+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:49.164509+0000","last_peered":"2024-09-10T20:20:49.164509+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:49.164509+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23450784,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2846","reported_seq":2896,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.342665+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:49.342665+0000","last_peered":"2024-09-10T20:20:49.342665+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:49.342665+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2846,"log_dups_size":0,"ondisk_log_size":2846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23286944,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2844,"num_write_kb":22749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2840","reported_seq":2880,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.369565+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:49.369565+0000","last_peered":"2024-09-10T20:20:49.369565+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:49.369565+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2840,"num_write_kb":22720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2812","reported_seq":2873,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.195195+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:49.195195+0000","last_peered":"2024-09-10T20:20:49.195195+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:49.195195+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2812,"log_dups_size":0,"ondisk_log_size":2812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22983840,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2810,"num_write_kb":22460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":853}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2839","reported_seq":2902,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.173088+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:49.173088+0000","last_peered":"2024-09-10T20:20:49.173088+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:49.173088+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2839,"log_dups_size":0,"ondisk_log_size":2839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213216,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":22684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2788","reported_seq":4298,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.368105+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:49.368105+0000","last_peered":"2024-09-10T20:20:49.368105+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:49.368105+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22767936,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2785,"num_write_kb":22257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1587}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2690","reported_seq":2766,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.897414+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:46.897414+0000","last_peered":"2024-09-10T20:20:46.897414+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:46.897414+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2690,"log_dups_size":0,"ondisk_log_size":2690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22017184,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":21502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2789","reported_seq":2843,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.272884+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:49.272884+0000","last_peered":"2024-09-10T20:20:49.272884+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:49.272884+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2789,"log_dups_size":0,"ondisk_log_size":2789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22802160,"num_objects":2784,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2854","reported_seq":2897,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.291913+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:49.291913+0000","last_peered":"2024-09-10T20:20:49.291913+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:49.291913+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23370320,"num_objects":2853,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":22823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2722","reported_seq":2796,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.957497+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:46.957497+0000","last_peered":"2024-09-10T20:20:46.957497+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:46.957497+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22288976,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":21767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2840","reported_seq":2890,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.130378+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:49.130378+0000","last_peered":"2024-09-10T20:20:49.130378+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:49.130378+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23229600,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2838,"num_write_kb":22700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":861}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2802","reported_seq":2880,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.872333+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:46.872333+0000","last_peered":"2024-09-10T20:20:46.872333+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:46.872333+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22926496,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":22397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2834","reported_seq":2897,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.156210+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:49.156210+0000","last_peered":"2024-09-10T20:20:49.156210+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:49.156210+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2834,"log_dups_size":0,"ondisk_log_size":2834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172256,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2832,"num_write_kb":22644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367485120,"num_objects":44864,"num_object_clones":0,"num_object_copies":179456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44891,"num_write_kb":359010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":605667328,"data_stored":605667328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44918,"ondisk_log_size":44918,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324140,"kb_used_data":179996,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94047700,"statfs":{"total":96636764160,"available":96304844800,"internally_reserved":0,"allocated":184315904,"data_stored":184139923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320004,"kb_used_data":176372,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94051836,"statfs":{"total":96636764160,"available":96309080064,"internally_reserved":0,"allocated":180604928,"data_stored":180428947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148536,"kb_used_data":54376,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94223304,"statfs":{"total":96636764160,"available":96484663296,"internally_reserved":0,"allocated":55681024,"data_stored":55505043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322752,"kb_used_data":178668,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94049088,"statfs":{"total":96636764160,"available":96306266112,"internally_reserved":0,"allocated":182956032,"data_stored":182780051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,7,4,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184446976,"data_stored":184446976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55705600,"data_stored":55705600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180752384,"data_stored":180752384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184762368,"data_stored":184762368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:52.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:52.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2806", 'reported_seq': 2846, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.365252+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:49.365252+0000', 'last_peered': '2024-09-10T20:20:49.365252+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:49.365252+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2806, 'log_dups_size': 0, 'ondisk_log_size': 2806, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22986752, 'num_objects': 2806, 'num_object_clones': 0, 'num_object_copies': 11224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2806, 'num_write_kb': 22448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 840}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:52.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2853", 'reported_seq': 2916, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.179992+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:49.179992+0000', 'last_peered': '2024-09-10T20:20:49.179992+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:49.179992+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2853, 'log_dups_size': 0, 'ondisk_log_size': 2853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23336096, 'num_objects': 2849, 'num_object_clones': 0, 'num_object_copies': 11396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2851, 'num_write_kb': 22804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2736", 'reported_seq': 2811, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:46.960318+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:46.960318+0000', 'last_peered': '2024-09-10T20:20:46.960318+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:46.960318+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2736, 'log_dups_size': 0, 'ondisk_log_size': 2736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22387280, 'num_objects': 2733, 'num_object_clones': 0, 'num_object_copies': 10932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2735, 'num_write_kb': 21870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 771}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2867", 'reported_seq': 2929, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.164509+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:49.164509+0000', 'last_peered': '2024-09-10T20:20:49.164509+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:49.164509+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2867, 'log_dups_size': 0, 'ondisk_log_size': 2867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23450784, 'num_objects': 2863, 'num_object_clones': 0, 'num_object_copies': 11452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2865, 'num_write_kb': 22909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2846", 'reported_seq': 2896, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.342665+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:49.342665+0000', 'last_peered': '2024-09-10T20:20:49.342665+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:49.342665+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2846, 'log_dups_size': 0, 'ondisk_log_size': 2846, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23286944, 'num_objects': 2843, 'num_object_clones': 0, 'num_object_copies': 11372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2844, 'num_write_kb': 22749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 863}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2840", 'reported_seq': 2880, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.369565+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:49.369565+0000', 'last_peered': '2024-09-10T20:20:49.369565+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:49.369565+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2840, 'log_dups_size': 0, 'ondisk_log_size': 2840, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23265280, 'num_objects': 2840, 'num_object_clones': 0, 'num_object_copies': 11360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2840, 'num_write_kb': 22720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 902}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:52.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2812", 'reported_seq': 2873, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.195195+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:49.195195+0000', 'last_peered': '2024-09-10T20:20:49.195195+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:49.195195+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2812, 'log_dups_size': 0, 'ondisk_log_size': 2812, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22983840, 'num_objects': 2806, 'num_object_clones': 0, 'num_object_copies': 11224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2810, 'num_write_kb': 22460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 853}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2839", 'reported_seq': 2902, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.173088+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:49.173088+0000', 'last_peered': '2024-09-10T20:20:49.173088+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:49.173088+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2839, 'log_dups_size': 0, 'ondisk_log_size': 2839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23213216, 'num_objects': 2834, 'num_object_clones': 0, 'num_object_copies': 11336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2837, 'num_write_kb': 22684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2788", 'reported_seq': 4298, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.368105+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:49.368105+0000', 'last_peered': '2024-09-10T20:20:49.368105+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:49.368105+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2788, 'log_dups_size': 0, 'ondisk_log_size': 2788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22767936, 'num_objects': 2780, 'num_object_clones': 0, 'num_object_copies': 11120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2785, 'num_write_kb': 22257, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 730, 'num_bytes_recovered': 5955584, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1587}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2690", 'reported_seq': 2766, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:46.897414+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:46.897414+0000', 'last_peered': '2024-09-10T20:20:46.897414+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:46.897414+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2690, 'log_dups_size': 0, 'ondisk_log_size': 2690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22017184, 'num_objects': 2688, 'num_object_clones': 0, 'num_object_copies': 10752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2688, 'num_write_kb': 21502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 785}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2789", 'reported_seq': 2843, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.272884+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:49.272884+0000', 'last_peered': '2024-09-10T20:20:49.272884+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:49.272884+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2789, 'log_dups_size': 0, 'ondisk_log_size': 2789, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22802160, 'num_objects': 2784, 'num_object_clones': 0, 'num_object_copies': 11136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2786, 'num_write_kb': 22276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 828}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:52.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2854", 'reported_seq': 2897, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.291913+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:49.291913+0000', 'last_peered': '2024-09-10T20:20:49.291913+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:49.291913+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2854, 'log_dups_size': 0, 'ondisk_log_size': 2854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23370320, 'num_objects': 2853, 'num_object_clones': 0, 'num_object_copies': 11412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2853, 'num_write_kb': 22823, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 875}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2722", 'reported_seq': 2796, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:46.957497+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:46.957497+0000', 'last_peered': '2024-09-10T20:20:46.957497+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:46.957497+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2722, 'log_dups_size': 0, 'ondisk_log_size': 2722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22288976, 'num_objects': 2721, 'num_object_clones': 0, 'num_object_copies': 10884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2721, 'num_write_kb': 21767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2840", 'reported_seq': 2890, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.130378+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:49.130378+0000', 'last_peered': '2024-09-10T20:20:49.130378+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:49.130378+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2840, 'log_dups_size': 0, 'ondisk_log_size': 2840, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23229600, 'num_objects': 2836, 'num_object_clones': 0, 'num_object_copies': 11344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2838, 'num_write_kb': 22700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 861}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2802", 'reported_seq': 2880, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:46.872333+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:46.872333+0000', 'last_peered': '2024-09-10T20:20:46.872333+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:46.872333+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2802, 'log_dups_size': 0, 'ondisk_log_size': 2802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22926496, 'num_objects': 2799, 'num_object_clones': 0, 'num_object_copies': 11196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2800, 'num_write_kb': 22397, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:52.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2834", 'reported_seq': 2897, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:49.156210+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:49.156210+0000', 'last_peered': '2024-09-10T20:20:49.156210+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:49.156210+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2834, 'log_dups_size': 0, 'ondisk_log_size': 2834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23172256, 'num_objects': 2829, 'num_object_clones': 0, 'num_object_copies': 11316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2832, 'num_write_kb': 22644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 850}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:52.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:52.780 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 71 20 46433 46413 5.10628 2.89844 0.033564 0.0305821 2024-09-10T20:20:52.953 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30754/179456 objects degraded (17.137%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":141,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44864,"data_bytes":367485120,"bytes_used":1142202368,"bytes_avail":385404854272,"bytes_total":386547056640,"degraded_objects":30754,"degraded_total":179456,"degraded_ratio":0.17137348430813124,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":110554,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":1327104,"num_keys_recovered":0,"write_bytes_sec":2788431,"read_op_per_sec":0,"write_op_per_sec":340},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:52.967 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:52.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:53.516 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:53.517 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:53.547 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-10T20:20:51.902177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367485120,"num_objects":44864,"num_object_clones":0,"num_object_copies":179456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44891,"num_write_kb":359010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44918,"ondisk_log_size":44918,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1115432,"kb_used_data":589412,"kb_used_omap":14,"kb_used_meta":525937,"kb_avail":376371928,"statfs":{"total":386547056640,"available":385404854272,"internally_reserved":0,"allocated":603557888,"data_stored":602853964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":538560054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,12,28,7,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33472512,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":32688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004064"},"pg_stats":[{"pgid":"1.f","version":"32'2806","reported_seq":2846,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.365252+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:49.365252+0000","last_peered":"2024-09-10T20:20:49.365252+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:49.365252+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2806,"log_dups_size":0,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22986752,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":22448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2853","reported_seq":2916,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.179992+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:49.179992+0000","last_peered":"2024-09-10T20:20:49.179992+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:49.179992+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336096,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2736","reported_seq":2811,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.960318+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:46.960318+0000","last_peered":"2024-09-10T20:20:46.960318+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:46.960318+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22387280,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":771}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2867","reported_seq":2929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.164509+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:49.164509+0000","last_peered":"2024-09-10T20:20:49.164509+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:49.164509+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23450784,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2846","reported_seq":2896,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.342665+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:49.342665+0000","last_peered":"2024-09-10T20:20:49.342665+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:49.342665+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2846,"log_dups_size":0,"ondisk_log_size":2846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23286944,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2844,"num_write_kb":22749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2840","reported_seq":2880,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.369565+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:49.369565+0000","last_peered":"2024-09-10T20:20:49.369565+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:49.369565+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2840,"num_write_kb":22720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2812","reported_seq":2873,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.195195+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:49.195195+0000","last_peered":"2024-09-10T20:20:49.195195+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:49.195195+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2812,"log_dups_size":0,"ondisk_log_size":2812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22983840,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2810,"num_write_kb":22460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":853}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2839","reported_seq":2902,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.173088+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:49.173088+0000","last_peered":"2024-09-10T20:20:49.173088+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:49.173088+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2839,"log_dups_size":0,"ondisk_log_size":2839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213216,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":22684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2788","reported_seq":4298,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.368105+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:49.368105+0000","last_peered":"2024-09-10T20:20:49.368105+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:49.368105+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22767936,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2785,"num_write_kb":22257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1587}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2690","reported_seq":2766,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.897414+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:46.897414+0000","last_peered":"2024-09-10T20:20:46.897414+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:46.897414+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2690,"log_dups_size":0,"ondisk_log_size":2690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22017184,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":21502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2789","reported_seq":2843,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.272884+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:49.272884+0000","last_peered":"2024-09-10T20:20:49.272884+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:49.272884+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2789,"log_dups_size":0,"ondisk_log_size":2789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22802160,"num_objects":2784,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2854","reported_seq":2897,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.291913+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:49.291913+0000","last_peered":"2024-09-10T20:20:49.291913+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:49.291913+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23370320,"num_objects":2853,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":22823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2722","reported_seq":2796,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.957497+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:46.957497+0000","last_peered":"2024-09-10T20:20:46.957497+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:46.957497+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22288976,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":21767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2840","reported_seq":2890,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.130378+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:49.130378+0000","last_peered":"2024-09-10T20:20:49.130378+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:49.130378+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23229600,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2838,"num_write_kb":22700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":861}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2802","reported_seq":2880,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:46.872333+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:46.872333+0000","last_peered":"2024-09-10T20:20:46.872333+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:46.872333+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22926496,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":22397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2834","reported_seq":2897,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:49.156210+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:49.156210+0000","last_peered":"2024-09-10T20:20:49.156210+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:49.156210+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2834,"log_dups_size":0,"ondisk_log_size":2834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172256,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2832,"num_write_kb":22644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367485120,"num_objects":44864,"num_object_clones":0,"num_object_copies":179456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44891,"num_write_kb":359010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":5955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":605667328,"data_stored":605667328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44918,"ondisk_log_size":44918,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324140,"kb_used_data":179996,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94047700,"statfs":{"total":96636764160,"available":96304844800,"internally_reserved":0,"allocated":184315904,"data_stored":184139923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320004,"kb_used_data":176372,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94051836,"statfs":{"total":96636764160,"available":96309080064,"internally_reserved":0,"allocated":180604928,"data_stored":180428947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148536,"kb_used_data":54376,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94223304,"statfs":{"total":96636764160,"available":96484663296,"internally_reserved":0,"allocated":55681024,"data_stored":55505043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322752,"kb_used_data":178668,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94049088,"statfs":{"total":96636764160,"available":96306266112,"internally_reserved":0,"allocated":182956032,"data_stored":182780051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,7,4,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184446976,"data_stored":184446976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55705600,"data_stored":55705600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180752384,"data_stored":180752384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184762368,"data_stored":184762368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:53.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 72 20 46766 46746 5.07149 2.60156 0.0464379 0.0307936 2024-09-10T20:20:54.780 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 73 20 47147 47127 5.04279 2.97656 0.0660797 0.0309667 2024-09-10T20:20:55.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 74 20 47572 47552 5.01951 3.32031 0.0319484 0.0311134 2024-09-10T20:20:56.550 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:56.780 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 75 20 47913 47893 4.98811 2.66406 0.0546016 0.031307 2024-09-10T20:20:57.112 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:57.112 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-10T20:20:55.903006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":383336640,"num_objects":46799,"num_object_clones":0,"num_object_copies":187196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46826,"num_write_kb":374490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46853,"ondisk_log_size":46853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1165548,"kb_used_data":621484,"kb_used_omap":14,"kb_used_meta":543985,"kb_avail":376321812,"statfs":{"total":386547056640,"available":385353535488,"internally_reserved":0,"allocated":636399616,"data_stored":635695692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":557041206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,9,13,6,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45826048,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":44752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1941504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003833"},"pg_stats":[{"pgid":"1.f","version":"32'2923","reported_seq":2963,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.369448+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:54.369448+0000","last_peered":"2024-09-10T20:20:54.369448+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:54.369448+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23945216,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":23384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2975","reported_seq":3038,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.085748+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:54.085748+0000","last_peered":"2024-09-10T20:20:54.085748+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:54.085748+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":0,"ondisk_log_size":2975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24335520,"num_objects":2971,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2857","reported_seq":2932,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:52.010207+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:52.010207+0000","last_peered":"2024-09-10T20:20:52.010207+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:52.010207+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2857,"log_dups_size":0,"ondisk_log_size":2857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23378512,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":22838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2978","reported_seq":3040,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.182560+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:54.182560+0000","last_peered":"2024-09-10T20:20:54.182560+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:54.182560+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360096,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2982","reported_seq":3033,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.338316+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:54.338316+0000","last_peered":"2024-09-10T20:20:54.338316+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:54.338316+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":0,"ondisk_log_size":2982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24401056,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2975","reported_seq":3016,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.285838+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:54.285838+0000","last_peered":"2024-09-10T20:20:54.285838+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:54.285838+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":0,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24371200,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2945","reported_seq":3009,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.114072+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:54.114072+0000","last_peered":"2024-09-10T20:20:54.114072+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:54.114072+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24073376,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2943,"num_write_kb":23524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":986}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2972","reported_seq":3035,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.169481+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:54.169481+0000","last_peered":"2024-09-10T20:20:54.169481+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:54.169481+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302752,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2970,"num_write_kb":23748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2915","reported_seq":4575,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.378252+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:54.378252+0000","last_peered":"2024-09-10T20:20:54.378252+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:54.378252+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23808320,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":23273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2815","reported_seq":2891,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:52.012262+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:52.012262+0000","last_peered":"2024-09-10T20:20:52.012262+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:52.012262+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":22502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2897","reported_seq":2951,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.270755+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:54.270755+0000","last_peered":"2024-09-10T20:20:54.270755+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:54.270755+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2897,"log_dups_size":0,"ondisk_log_size":2897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23686896,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":23140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2964","reported_seq":3007,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.344391+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:54.344391+0000","last_peered":"2024-09-10T20:20:54.344391+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:54.344391+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2964,"log_dups_size":0,"ondisk_log_size":2964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24271440,"num_objects":2963,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2963,"num_write_kb":23703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":985}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2812","reported_seq":2886,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:51.738413+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:51.738413+0000","last_peered":"2024-09-10T20:20:51.738413+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:51.738413+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2812,"log_dups_size":0,"ondisk_log_size":2812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23026256,"num_objects":2811,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":22487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":863}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2958","reported_seq":3007,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.215163+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:54.215163+0000","last_peered":"2024-09-10T20:20:54.215163+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:54.215163+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":0,"ondisk_log_size":2958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24196256,"num_objects":2954,"num_object_clones":0,"num_object_copies":11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2956,"num_write_kb":23644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2933","reported_seq":3011,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:51.979652+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:51.979652+0000","last_peered":"2024-09-10T20:20:51.979652+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:51.979652+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23999648,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":23445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2952","reported_seq":3015,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.138377+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:54.138377+0000","last_peered":"2024-09-10T20:20:54.138377+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:54.138377+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":0,"ondisk_log_size":2952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138912,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":383336640,"num_objects":46799,"num_object_clones":0,"num_object_copies":187196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46826,"num_write_kb":374490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638382080,"data_stored":638382080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46853,"ondisk_log_size":46853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354244,"kb_used_data":187892,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94017596,"statfs":{"total":96636764160,"available":96274018304,"internally_reserved":0,"allocated":192401408,"data_stored":192225427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":170323664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332216,"kb_used_data":184424,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94039624,"statfs":{"total":96636764160,"available":96296574976,"internally_reserved":0,"allocated":188850176,"data_stored":188674195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144320,"kb_used_data":62640,"kb_used_omap":4,"kb_used_meta":81659,"kb_avail":94227520,"statfs":{"total":96636764160,"available":96488980480,"internally_reserved":0,"allocated":64143360,"data_stored":63967379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83619538},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334768,"kb_used_data":186528,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94037072,"statfs":{"total":96636764160,"available":96293961728,"internally_reserved":0,"allocated":191004672,"data_stored":190828691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,7,4,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192348160,"data_stored":192348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":64360448,"data_stored":64360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":189034496,"data_stored":189034496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192638976,"data_stored":192638976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:57.133 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:57.746 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:57.778 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-10T20:20:55.903006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":383336640,"num_objects":46799,"num_object_clones":0,"num_object_copies":187196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46826,"num_write_kb":374490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46853,"ondisk_log_size":46853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1165548,"kb_used_data":621484,"kb_used_omap":14,"kb_used_meta":543985,"kb_avail":376321812,"statfs":{"total":386547056640,"available":385353535488,"internally_reserved":0,"allocated":636399616,"data_stored":635695692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":557041206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,9,13,6,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45826048,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":44752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1941504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003833"},"pg_stats":[{"pgid":"1.f","version":"32'2923","reported_seq":2963,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.369448+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:54.369448+0000","last_peered":"2024-09-10T20:20:54.369448+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:54.369448+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23945216,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":23384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2975","reported_seq":3038,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.085748+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:54.085748+0000","last_peered":"2024-09-10T20:20:54.085748+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:54.085748+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":0,"ondisk_log_size":2975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24335520,"num_objects":2971,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2857","reported_seq":2932,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:52.010207+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:52.010207+0000","last_peered":"2024-09-10T20:20:52.010207+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:52.010207+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2857,"log_dups_size":0,"ondisk_log_size":2857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23378512,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":22838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2978","reported_seq":3040,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.182560+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:54.182560+0000","last_peered":"2024-09-10T20:20:54.182560+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:54.182560+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360096,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2982","reported_seq":3033,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.338316+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:54.338316+0000","last_peered":"2024-09-10T20:20:54.338316+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:54.338316+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":0,"ondisk_log_size":2982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24401056,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2975","reported_seq":3016,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.285838+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:54.285838+0000","last_peered":"2024-09-10T20:20:54.285838+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:54.285838+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":0,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24371200,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2945","reported_seq":3009,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.114072+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:54.114072+0000","last_peered":"2024-09-10T20:20:54.114072+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:54.114072+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24073376,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2943,"num_write_kb":23524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":986}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2972","reported_seq":3035,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.169481+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:54.169481+0000","last_peered":"2024-09-10T20:20:54.169481+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:54.169481+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302752,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2970,"num_write_kb":23748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2915","reported_seq":4575,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.378252+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:54.378252+0000","last_peered":"2024-09-10T20:20:54.378252+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:54.378252+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23808320,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":23273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2815","reported_seq":2891,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:52.012262+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:52.012262+0000","last_peered":"2024-09-10T20:20:52.012262+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:52.012262+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":22502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2897","reported_seq":2951,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.270755+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:54.270755+0000","last_peered":"2024-09-10T20:20:54.270755+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:54.270755+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2897,"log_dups_size":0,"ondisk_log_size":2897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23686896,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":23140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2964","reported_seq":3007,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.344391+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:54.344391+0000","last_peered":"2024-09-10T20:20:54.344391+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:54.344391+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2964,"log_dups_size":0,"ondisk_log_size":2964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24271440,"num_objects":2963,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2963,"num_write_kb":23703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":985}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2812","reported_seq":2886,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:51.738413+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:51.738413+0000","last_peered":"2024-09-10T20:20:51.738413+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:51.738413+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2812,"log_dups_size":0,"ondisk_log_size":2812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23026256,"num_objects":2811,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":22487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":863}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2958","reported_seq":3007,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.215163+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:54.215163+0000","last_peered":"2024-09-10T20:20:54.215163+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:54.215163+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":0,"ondisk_log_size":2958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24196256,"num_objects":2954,"num_object_clones":0,"num_object_copies":11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2956,"num_write_kb":23644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2933","reported_seq":3011,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:51.979652+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:51.979652+0000","last_peered":"2024-09-10T20:20:51.979652+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:51.979652+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23999648,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":23445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2952","reported_seq":3015,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.138377+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:54.138377+0000","last_peered":"2024-09-10T20:20:54.138377+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:54.138377+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":0,"ondisk_log_size":2952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138912,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":383336640,"num_objects":46799,"num_object_clones":0,"num_object_copies":187196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46826,"num_write_kb":374490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638382080,"data_stored":638382080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46853,"ondisk_log_size":46853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354244,"kb_used_data":187892,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94017596,"statfs":{"total":96636764160,"available":96274018304,"internally_reserved":0,"allocated":192401408,"data_stored":192225427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":170323664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332216,"kb_used_data":184424,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94039624,"statfs":{"total":96636764160,"available":96296574976,"internally_reserved":0,"allocated":188850176,"data_stored":188674195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144320,"kb_used_data":62640,"kb_used_omap":4,"kb_used_meta":81659,"kb_avail":94227520,"statfs":{"total":96636764160,"available":96488980480,"internally_reserved":0,"allocated":64143360,"data_stored":63967379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83619538},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334768,"kb_used_data":186528,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94037072,"statfs":{"total":96636764160,"available":96293961728,"internally_reserved":0,"allocated":191004672,"data_stored":190828691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,7,4,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192348160,"data_stored":192348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":64360448,"data_stored":64360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":189034496,"data_stored":189034496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192638976,"data_stored":192638976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:57.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:20:57.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2923", 'reported_seq': 2963, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.369448+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:54.369448+0000', 'last_peered': '2024-09-10T20:20:54.369448+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:54.369448+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2923, 'log_dups_size': 0, 'ondisk_log_size': 2923, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23945216, 'num_objects': 2923, 'num_object_clones': 0, 'num_object_copies': 11692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2923, 'num_write_kb': 23384, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 957}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2975", 'reported_seq': 3038, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.085748+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:54.085748+0000', 'last_peered': '2024-09-10T20:20:54.085748+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:54.085748+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2975, 'log_dups_size': 0, 'ondisk_log_size': 2975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24335520, 'num_objects': 2971, 'num_object_clones': 0, 'num_object_copies': 11884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2973, 'num_write_kb': 23780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2857", 'reported_seq': 2932, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:52.010207+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:52.010207+0000', 'last_peered': '2024-09-10T20:20:52.010207+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:52.010207+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2857, 'log_dups_size': 0, 'ondisk_log_size': 2857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23378512, 'num_objects': 2854, 'num_object_clones': 0, 'num_object_copies': 11416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2856, 'num_write_kb': 22838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 892}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2978", 'reported_seq': 3040, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.182560+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:54.182560+0000', 'last_peered': '2024-09-10T20:20:54.182560+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:54.182560+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2978, 'log_dups_size': 0, 'ondisk_log_size': 2978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24360096, 'num_objects': 2974, 'num_object_clones': 0, 'num_object_copies': 11896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2976, 'num_write_kb': 23797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 971}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2982", 'reported_seq': 3033, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.338316+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:54.338316+0000', 'last_peered': '2024-09-10T20:20:54.338316+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:54.338316+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2982, 'log_dups_size': 0, 'ondisk_log_size': 2982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24401056, 'num_objects': 2979, 'num_object_clones': 0, 'num_object_copies': 11916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2980, 'num_write_kb': 23837, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 999}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:20:57.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2975", 'reported_seq': 3016, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.285838+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:54.285838+0000', 'last_peered': '2024-09-10T20:20:54.285838+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:54.285838+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2975, 'log_dups_size': 0, 'ondisk_log_size': 2975, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24371200, 'num_objects': 2975, 'num_object_clones': 0, 'num_object_copies': 11900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2975, 'num_write_kb': 23800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1037}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:57.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:20:57.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2945", 'reported_seq': 3009, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.114072+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:54.114072+0000', 'last_peered': '2024-09-10T20:20:54.114072+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:54.114072+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2945, 'log_dups_size': 0, 'ondisk_log_size': 2945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24073376, 'num_objects': 2939, 'num_object_clones': 0, 'num_object_copies': 11756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2943, 'num_write_kb': 23524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 986}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:57.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:20:57.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2972", 'reported_seq': 3035, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.169481+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:54.169481+0000', 'last_peered': '2024-09-10T20:20:54.169481+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:54.169481+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2972, 'log_dups_size': 0, 'ondisk_log_size': 2972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24302752, 'num_objects': 2967, 'num_object_clones': 0, 'num_object_copies': 11868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2970, 'num_write_kb': 23748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 924}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:57.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:20:57.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2915", 'reported_seq': 4575, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.378252+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:54.378252+0000', 'last_peered': '2024-09-10T20:20:54.378252+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:54.378252+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2915, 'log_dups_size': 0, 'ondisk_log_size': 2915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23808320, 'num_objects': 2907, 'num_object_clones': 0, 'num_object_copies': 11628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2912, 'num_write_kb': 23273, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 805, 'num_bytes_recovered': 6569984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1789}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:57.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:20:57.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2815", 'reported_seq': 2891, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:52.012262+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:52.012262+0000', 'last_peered': '2024-09-10T20:20:52.012262+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:52.012262+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2815, 'log_dups_size': 0, 'ondisk_log_size': 2815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23041184, 'num_objects': 2813, 'num_object_clones': 0, 'num_object_copies': 11252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2813, 'num_write_kb': 22502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 910}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:57.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:20:57.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2897", 'reported_seq': 2951, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.270755+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:54.270755+0000', 'last_peered': '2024-09-10T20:20:54.270755+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:54.270755+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2897, 'log_dups_size': 0, 'ondisk_log_size': 2897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23686896, 'num_objects': 2892, 'num_object_clones': 0, 'num_object_copies': 11568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2894, 'num_write_kb': 23140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 936}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:57.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:20:57.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2964", 'reported_seq': 3007, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.344391+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:54.344391+0000', 'last_peered': '2024-09-10T20:20:54.344391+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:54.344391+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2964, 'log_dups_size': 0, 'ondisk_log_size': 2964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24271440, 'num_objects': 2963, 'num_object_clones': 0, 'num_object_copies': 11852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2963, 'num_write_kb': 23703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:57.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:20:57.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2812", 'reported_seq': 2886, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:51.738413+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:51.738413+0000', 'last_peered': '2024-09-10T20:20:51.738413+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:51.738413+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2812, 'log_dups_size': 0, 'ondisk_log_size': 2812, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23026256, 'num_objects': 2811, 'num_object_clones': 0, 'num_object_copies': 11244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2811, 'num_write_kb': 22487, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 863}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:57.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:20:57.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2958", 'reported_seq': 3007, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.215163+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:54.215163+0000', 'last_peered': '2024-09-10T20:20:54.215163+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:54.215163+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2958, 'log_dups_size': 0, 'ondisk_log_size': 2958, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24196256, 'num_objects': 2954, 'num_object_clones': 0, 'num_object_copies': 11816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2956, 'num_write_kb': 23644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 979}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:20:57.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:20:57.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2933", 'reported_seq': 3011, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:51.979652+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:51.979652+0000', 'last_peered': '2024-09-10T20:20:51.979652+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:51.979652+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2933, 'log_dups_size': 0, 'ondisk_log_size': 2933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23999648, 'num_objects': 2930, 'num_object_clones': 0, 'num_object_copies': 11720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2931, 'num_write_kb': 23445, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:20:57.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:20:57.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2952", 'reported_seq': 3015, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:54.138377+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:54.138377+0000', 'last_peered': '2024-09-10T20:20:54.138377+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:54.138377+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2952, 'log_dups_size': 0, 'ondisk_log_size': 2952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24138912, 'num_objects': 2947, 'num_object_clones': 0, 'num_object_copies': 11788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2950, 'num_write_kb': 23588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:20:57.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:20:57.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 76 20 48282 48262 4.9604 2.88281 0.0780812 0.0314829 2024-09-10T20:20:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:58.516 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30679/189000 objects degraded (16.232%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":147,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47250,"data_bytes":387031232,"bytes_used":1218805760,"bytes_avail":385328250880,"bytes_total":386547056640,"degraded_objects":30679,"degraded_total":189000,"degraded_ratio":0.16232275132275131,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":100319,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"write_bytes_sec":3068968,"read_op_per_sec":0,"write_op_per_sec":374},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:20:58.537 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:20:58.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:20:58.781 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 77 20 48624 48604 4.93068 2.67188 0.0371137 0.0316739 2024-09-10T20:20:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:20:59.088 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:20:59.108 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-10T20:20:57.903495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":387031232,"num_objects":47250,"num_object_clones":0,"num_object_copies":189000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47277,"num_write_kb":378098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47304,"ondisk_log_size":47304,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1190240,"kb_used_data":636448,"kb_used_omap":14,"kb_used_meta":553713,"kb_avail":376297120,"statfs":{"total":386547056640,"available":385328250880,"internally_reserved":0,"allocated":651722752,"data_stored":651018828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":567002678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,18,19,15,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36839424,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003845"},"pg_stats":[{"pgid":"1.f","version":"32'2923","reported_seq":2963,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.369448+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:54.369448+0000","last_peered":"2024-09-10T20:20:54.369448+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:54.369448+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23945216,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":23384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2975","reported_seq":3038,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.085748+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:54.085748+0000","last_peered":"2024-09-10T20:20:54.085748+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:54.085748+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":0,"ondisk_log_size":2975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24335520,"num_objects":2971,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2981","reported_seq":3056,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:57.030732+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:57.030732+0000","last_peered":"2024-09-10T20:20:57.030732+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:57.030732+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24394320,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2978","reported_seq":3040,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.182560+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:54.182560+0000","last_peered":"2024-09-10T20:20:54.182560+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:54.182560+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360096,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2982","reported_seq":3033,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.338316+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:54.338316+0000","last_peered":"2024-09-10T20:20:54.338316+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:54.338316+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":0,"ondisk_log_size":2982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24401056,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2975","reported_seq":3016,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.285838+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:54.285838+0000","last_peered":"2024-09-10T20:20:54.285838+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:54.285838+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":0,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24371200,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2945","reported_seq":3009,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.114072+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:54.114072+0000","last_peered":"2024-09-10T20:20:54.114072+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:54.114072+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24073376,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2943,"num_write_kb":23524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":986}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2972","reported_seq":3035,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.169481+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:54.169481+0000","last_peered":"2024-09-10T20:20:54.169481+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:54.169481+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302752,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2970,"num_write_kb":23748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2915","reported_seq":4575,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.378252+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:54.378252+0000","last_peered":"2024-09-10T20:20:54.378252+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:54.378252+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23808320,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":23273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2914","reported_seq":2990,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:56.955669+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:56.955669+0000","last_peered":"2024-09-10T20:20:56.955669+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:56.955669+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2914,"log_dups_size":0,"ondisk_log_size":2914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23852192,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":23294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2897","reported_seq":2951,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.270755+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:54.270755+0000","last_peered":"2024-09-10T20:20:54.270755+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:54.270755+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2897,"log_dups_size":0,"ondisk_log_size":2897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23686896,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":23140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2964","reported_seq":3007,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.344391+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:54.344391+0000","last_peered":"2024-09-10T20:20:54.344391+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:54.344391+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2964,"log_dups_size":0,"ondisk_log_size":2964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24271440,"num_objects":2963,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2963,"num_write_kb":23703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":985}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2922","reported_seq":2996,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:57.031995+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:57.031995+0000","last_peered":"2024-09-10T20:20:57.031995+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:57.031995+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23927376,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2921,"num_write_kb":23367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2958","reported_seq":3007,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.215163+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:54.215163+0000","last_peered":"2024-09-10T20:20:54.215163+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:54.215163+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":0,"ondisk_log_size":2958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24196256,"num_objects":2954,"num_object_clones":0,"num_object_copies":11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2956,"num_write_kb":23644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3051","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:56.920994+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:56.920994+0000","last_peered":"2024-09-10T20:20:56.920994+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:56.920994+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24966304,"num_objects":3048,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":24389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2952","reported_seq":3015,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:54.138377+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:54.138377+0000","last_peered":"2024-09-10T20:20:54.138377+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:54.138377+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":0,"ondisk_log_size":2952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138912,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":387031232,"num_objects":47250,"num_object_clones":0,"num_object_copies":189000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47277,"num_write_kb":378098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":6569984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":653914112,"data_stored":653914112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47304,"ondisk_log_size":47304,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354244,"kb_used_data":187892,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94017596,"statfs":{"total":96636764160,"available":96274018304,"internally_reserved":0,"allocated":192401408,"data_stored":192225427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":170323664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345168,"kb_used_data":191808,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94026672,"statfs":{"total":96636764160,"available":96283312128,"internally_reserved":0,"allocated":196411392,"data_stored":196235411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":156060,"kb_used_data":70220,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94215780,"statfs":{"total":96636764160,"available":96476958720,"internally_reserved":0,"allocated":71905280,"data_stored":71729299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87879378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,9,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334768,"kb_used_data":186528,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94037072,"statfs":{"total":96636764160,"available":96293961728,"internally_reserved":0,"allocated":191004672,"data_stored":190828691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,7,4,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192348160,"data_stored":192348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":72220672,"data_stored":72220672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196706304,"data_stored":196706304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192638976,"data_stored":192638976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:20:59.781 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 78 20 49033 49013 4.90843 3.19531 0.0335862 0.031816 2024-09-10T20:21:00.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 79 20 49464 49444 4.88885 3.36719 0.0736988 0.0319481 2024-09-10T20:21:01.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:21:01.779647+0000 min lat: 0.00230373 max lat: 2.76361 avg lat: 0.0320773 2024-09-10T20:21:01.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:21:01.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 80 20 49883 49863 4.86866 3.27344 0.0361942 0.0320773 2024-09-10T20:21:02.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:02.650 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:02.651 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:02.680 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-10T20:21:01.904968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398532800,"num_objects":48654,"num_object_clones":0,"num_object_copies":194616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48681,"num_write_kb":389330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48708,"ondisk_log_size":48708,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1220584,"kb_used_data":651496,"kb_used_omap":14,"kb_used_meta":569009,"kb_avail":376266776,"statfs":{"total":386547056640,"available":385297178624,"internally_reserved":0,"allocated":667131904,"data_stored":666427980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":582665782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,11,16,13,4,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31047680,"num_objects":3790,"num_object_clones":0,"num_object_copies":15160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3790,"num_write_kb":30320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004435"},"pg_stats":[{"pgid":"1.f","version":"32'3027","reported_seq":3068,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.319801+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:59.319801+0000","last_peered":"2024-09-10T20:20:59.319801+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:59.319801+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3027,"log_dups_size":0,"ondisk_log_size":3027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24797184,"num_objects":3027,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":24216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3084","reported_seq":3146,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.176627+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:59.176627+0000","last_peered":"2024-09-10T20:20:59.176627+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:59.176627+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3084,"log_dups_size":0,"ondisk_log_size":3084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3082,"num_write_kb":24652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2981","reported_seq":3056,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:57.030732+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:57.030732+0000","last_peered":"2024-09-10T20:20:57.030732+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:57.030732+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24394320,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3111","reported_seq":3173,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.189384+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:59.189384+0000","last_peered":"2024-09-10T20:20:59.189384+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:59.189384+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25449632,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3109,"num_write_kb":24861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3089","reported_seq":3140,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.333279+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:59.333279+0000","last_peered":"2024-09-10T20:20:59.333279+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:59.333279+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25277600,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3087,"num_write_kb":24693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3088","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.220247+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:59.220247+0000","last_peered":"2024-09-10T20:20:59.220247+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:59.220247+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":24704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3076","reported_seq":3139,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.221810+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:59.221810+0000","last_peered":"2024-09-10T20:20:59.221810+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:59.221810+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3076,"log_dups_size":0,"ondisk_log_size":3076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25146528,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3074,"num_write_kb":24572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1117}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3095","reported_seq":3157,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.147899+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:59.147899+0000","last_peered":"2024-09-10T20:20:59.147899+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:59.147899+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310368,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3020","reported_seq":4846,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.372747+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:59.372747+0000","last_peered":"2024-09-10T20:20:59.372747+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:59.372747+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24668480,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2914","reported_seq":2990,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:56.955669+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:56.955669+0000","last_peered":"2024-09-10T20:20:56.955669+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:56.955669+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2914,"log_dups_size":0,"ondisk_log_size":2914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23852192,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":23294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3025","reported_seq":3077,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.376645+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:59.376645+0000","last_peered":"2024-09-10T20:20:59.376645+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:59.376645+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3025,"log_dups_size":0,"ondisk_log_size":3025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24735472,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3022,"num_write_kb":24164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3093","reported_seq":3136,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.386003+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:59.386003+0000","last_peered":"2024-09-10T20:20:59.386003+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:59.386003+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25328208,"num_objects":3092,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1114}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2922","reported_seq":2996,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:57.031995+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:57.031995+0000","last_peered":"2024-09-10T20:20:57.031995+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:57.031995+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23927376,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2921,"num_write_kb":23367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3068","reported_seq":3118,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.287831+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:59.287831+0000","last_peered":"2024-09-10T20:20:59.287831+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:59.287831+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25097376,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":24524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3051","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:56.920994+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:56.920994+0000","last_peered":"2024-09-10T20:20:56.920994+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:56.920994+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24966304,"num_objects":3048,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":24389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3064","reported_seq":3127,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.126137+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:59.126137+0000","last_peered":"2024-09-10T20:20:59.126137+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:59.126137+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3064,"log_dups_size":0,"ondisk_log_size":3064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3062,"num_write_kb":24484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":398532800,"num_objects":48654,"num_object_clones":0,"num_object_copies":194616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48681,"num_write_kb":389330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":669011968,"data_stored":669011968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48708,"ondisk_log_size":48708,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371316,"kb_used_data":195236,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94000524,"statfs":{"total":96636764160,"available":96256536576,"internally_reserved":0,"allocated":199921664,"data_stored":199745683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345168,"kb_used_data":191808,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94026672,"statfs":{"total":96636764160,"available":96283312128,"internally_reserved":0,"allocated":196411392,"data_stored":196235411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":156060,"kb_used_data":70220,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94215780,"statfs":{"total":96636764160,"available":96476958720,"internally_reserved":0,"allocated":71905280,"data_stored":71729299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87879378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,9,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348040,"kb_used_data":194232,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94023800,"statfs":{"total":96636764160,"available":96280371200,"internally_reserved":0,"allocated":198893568,"data_stored":198717587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199897088,"data_stored":199897088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":72220672,"data_stored":72220672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196706304,"data_stored":196706304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200187904,"data_stored":200187904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:02.681 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:02.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 81 20 50257 50237 4.84462 2.92188 0.057242 0.0322415 2024-09-10T20:21:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:03.296 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-10T20:21:01.904968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398532800,"num_objects":48654,"num_object_clones":0,"num_object_copies":194616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48681,"num_write_kb":389330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48708,"ondisk_log_size":48708,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1220584,"kb_used_data":651496,"kb_used_omap":14,"kb_used_meta":569009,"kb_avail":376266776,"statfs":{"total":386547056640,"available":385297178624,"internally_reserved":0,"allocated":667131904,"data_stored":666427980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":582665782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,11,16,13,4,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31047680,"num_objects":3790,"num_object_clones":0,"num_object_copies":15160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3790,"num_write_kb":30320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004435"},"pg_stats":[{"pgid":"1.f","version":"32'3027","reported_seq":3068,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.319801+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:59.319801+0000","last_peered":"2024-09-10T20:20:59.319801+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:59.319801+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3027,"log_dups_size":0,"ondisk_log_size":3027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24797184,"num_objects":3027,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":24216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3084","reported_seq":3146,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.176627+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:59.176627+0000","last_peered":"2024-09-10T20:20:59.176627+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:59.176627+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3084,"log_dups_size":0,"ondisk_log_size":3084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3082,"num_write_kb":24652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2981","reported_seq":3056,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:57.030732+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:20:57.030732+0000","last_peered":"2024-09-10T20:20:57.030732+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:20:57.030732+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24394320,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3111","reported_seq":3173,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.189384+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:59.189384+0000","last_peered":"2024-09-10T20:20:59.189384+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:59.189384+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25449632,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3109,"num_write_kb":24861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3089","reported_seq":3140,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.333279+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:59.333279+0000","last_peered":"2024-09-10T20:20:59.333279+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:59.333279+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25277600,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3087,"num_write_kb":24693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3088","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.220247+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:59.220247+0000","last_peered":"2024-09-10T20:20:59.220247+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:59.220247+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":24704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3076","reported_seq":3139,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.221810+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:59.221810+0000","last_peered":"2024-09-10T20:20:59.221810+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:59.221810+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3076,"log_dups_size":0,"ondisk_log_size":3076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25146528,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3074,"num_write_kb":24572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1117}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3095","reported_seq":3157,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.147899+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:59.147899+0000","last_peered":"2024-09-10T20:20:59.147899+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:59.147899+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310368,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3020","reported_seq":4846,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.372747+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:59.372747+0000","last_peered":"2024-09-10T20:20:59.372747+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:59.372747+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24668480,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2914","reported_seq":2990,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:56.955669+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:20:56.955669+0000","last_peered":"2024-09-10T20:20:56.955669+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:20:56.955669+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2914,"log_dups_size":0,"ondisk_log_size":2914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23852192,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":23294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3025","reported_seq":3077,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.376645+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:59.376645+0000","last_peered":"2024-09-10T20:20:59.376645+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:59.376645+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3025,"log_dups_size":0,"ondisk_log_size":3025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24735472,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3022,"num_write_kb":24164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3093","reported_seq":3136,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.386003+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:59.386003+0000","last_peered":"2024-09-10T20:20:59.386003+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:59.386003+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25328208,"num_objects":3092,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1114}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2922","reported_seq":2996,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:57.031995+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:20:57.031995+0000","last_peered":"2024-09-10T20:20:57.031995+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:20:57.031995+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23927376,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2921,"num_write_kb":23367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3068","reported_seq":3118,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.287831+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:59.287831+0000","last_peered":"2024-09-10T20:20:59.287831+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:59.287831+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25097376,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":24524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3051","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:56.920994+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:20:56.920994+0000","last_peered":"2024-09-10T20:20:56.920994+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:20:56.920994+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24966304,"num_objects":3048,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":24389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3064","reported_seq":3127,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.126137+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:59.126137+0000","last_peered":"2024-09-10T20:20:59.126137+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:59.126137+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3064,"log_dups_size":0,"ondisk_log_size":3064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3062,"num_write_kb":24484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":398532800,"num_objects":48654,"num_object_clones":0,"num_object_copies":194616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48681,"num_write_kb":389330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":669011968,"data_stored":669011968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48708,"ondisk_log_size":48708,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371316,"kb_used_data":195236,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94000524,"statfs":{"total":96636764160,"available":96256536576,"internally_reserved":0,"allocated":199921664,"data_stored":199745683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345168,"kb_used_data":191808,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94026672,"statfs":{"total":96636764160,"available":96283312128,"internally_reserved":0,"allocated":196411392,"data_stored":196235411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":156060,"kb_used_data":70220,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94215780,"statfs":{"total":96636764160,"available":96476958720,"internally_reserved":0,"allocated":71905280,"data_stored":71729299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87879378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,9,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348040,"kb_used_data":194232,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94023800,"statfs":{"total":96636764160,"available":96280371200,"internally_reserved":0,"allocated":198893568,"data_stored":198717587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199897088,"data_stored":199897088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":72220672,"data_stored":72220672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196706304,"data_stored":196706304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200187904,"data_stored":200187904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:03.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:03.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3027", 'reported_seq': 3068, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.319801+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:20:59.319801+0000', 'last_peered': '2024-09-10T20:20:59.319801+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:20:59.319801+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3027, 'log_dups_size': 0, 'ondisk_log_size': 3027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24797184, 'num_objects': 3027, 'num_object_clones': 0, 'num_object_copies': 12108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3027, 'num_write_kb': 24216, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1061}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:03.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:21:03.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3084", 'reported_seq': 3146, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.176627+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:20:59.176627+0000', 'last_peered': '2024-09-10T20:20:59.176627+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:20:59.176627+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3084, 'log_dups_size': 0, 'ondisk_log_size': 3084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25228448, 'num_objects': 3080, 'num_object_clones': 0, 'num_object_copies': 12320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3082, 'num_write_kb': 24652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:03.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:03.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2981", 'reported_seq': 3056, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:57.030732+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:20:57.030732+0000', 'last_peered': '2024-09-10T20:20:57.030732+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:20:57.030732+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2981, 'log_dups_size': 0, 'ondisk_log_size': 2981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24394320, 'num_objects': 2978, 'num_object_clones': 0, 'num_object_copies': 11912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2980, 'num_write_kb': 23830, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1016}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3111", 'reported_seq': 3173, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.189384+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:20:59.189384+0000', 'last_peered': '2024-09-10T20:20:59.189384+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:20:59.189384+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3111, 'log_dups_size': 0, 'ondisk_log_size': 3111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25449632, 'num_objects': 3107, 'num_object_clones': 0, 'num_object_copies': 12428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3109, 'num_write_kb': 24861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3089", 'reported_seq': 3140, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.333279+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:20:59.333279+0000', 'last_peered': '2024-09-10T20:20:59.333279+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:20:59.333279+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3089, 'log_dups_size': 0, 'ondisk_log_size': 3089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25277600, 'num_objects': 3086, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3087, 'num_write_kb': 24693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1106}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3088", 'reported_seq': 3129, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.220247+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:20:59.220247+0000', 'last_peered': '2024-09-10T20:20:59.220247+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:20:59.220247+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3088, 'log_dups_size': 0, 'ondisk_log_size': 3088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25296896, 'num_objects': 3088, 'num_object_clones': 0, 'num_object_copies': 12352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3088, 'num_write_kb': 24704, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1150}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3076", 'reported_seq': 3139, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.221810+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:20:59.221810+0000', 'last_peered': '2024-09-10T20:20:59.221810+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:20:59.221810+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3076, 'log_dups_size': 0, 'ondisk_log_size': 3076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25146528, 'num_objects': 3070, 'num_object_clones': 0, 'num_object_copies': 12280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3074, 'num_write_kb': 24572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1117}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:03.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3095", 'reported_seq': 3157, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.147899+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:20:59.147899+0000', 'last_peered': '2024-09-10T20:20:59.147899+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:20:59.147899+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3095, 'log_dups_size': 0, 'ondisk_log_size': 3095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25310368, 'num_objects': 3090, 'num_object_clones': 0, 'num_object_copies': 12360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3093, 'num_write_kb': 24732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1047}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3020", 'reported_seq': 4846, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.372747+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:20:59.372747+0000', 'last_peered': '2024-09-10T20:20:59.372747+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:20:59.372747+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3020, 'log_dups_size': 0, 'ondisk_log_size': 3020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24668480, 'num_objects': 3012, 'num_object_clones': 0, 'num_object_copies': 12048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3017, 'num_write_kb': 24113, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 888, 'num_bytes_recovered': 7249920, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1977}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2914", 'reported_seq': 2990, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:56.955669+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:20:56.955669+0000', 'last_peered': '2024-09-10T20:20:56.955669+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:20:56.955669+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2914, 'log_dups_size': 0, 'ondisk_log_size': 2914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23852192, 'num_objects': 2912, 'num_object_clones': 0, 'num_object_copies': 11648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2912, 'num_write_kb': 23294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1009}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3025", 'reported_seq': 3077, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.376645+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:20:59.376645+0000', 'last_peered': '2024-09-10T20:20:59.376645+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:20:59.376645+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3025, 'log_dups_size': 0, 'ondisk_log_size': 3025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24735472, 'num_objects': 3020, 'num_object_clones': 0, 'num_object_copies': 12080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3022, 'num_write_kb': 24164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3093", 'reported_seq': 3136, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.386003+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:20:59.386003+0000', 'last_peered': '2024-09-10T20:20:59.386003+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:20:59.386003+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3093, 'log_dups_size': 0, 'ondisk_log_size': 3093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25328208, 'num_objects': 3092, 'num_object_clones': 0, 'num_object_copies': 12368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3092, 'num_write_kb': 24735, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2922", 'reported_seq': 2996, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:57.031995+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:20:57.031995+0000', 'last_peered': '2024-09-10T20:20:57.031995+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:20:57.031995+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 2922, 'log_dups_size': 0, 'ondisk_log_size': 2922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23927376, 'num_objects': 2921, 'num_object_clones': 0, 'num_object_copies': 11684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2921, 'num_write_kb': 23367, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:03.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:03.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3068", 'reported_seq': 3118, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.287831+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:20:59.287831+0000', 'last_peered': '2024-09-10T20:20:59.287831+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:20:59.287831+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3068, 'log_dups_size': 0, 'ondisk_log_size': 3068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25097376, 'num_objects': 3064, 'num_object_clones': 0, 'num_object_copies': 12256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3066, 'num_write_kb': 24524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:03.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:03.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3051", 'reported_seq': 3129, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:56.920994+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:20:56.920994+0000', 'last_peered': '2024-09-10T20:20:56.920994+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:20:56.920994+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3051, 'log_dups_size': 0, 'ondisk_log_size': 3051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24966304, 'num_objects': 3048, 'num_object_clones': 0, 'num_object_copies': 12192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3049, 'num_write_kb': 24389, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:03.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:03.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3064", 'reported_seq': 3127, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:20:59.126137+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:20:59.126137+0000', 'last_peered': '2024-09-10T20:20:59.126137+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:20:59.126137+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3064, 'log_dups_size': 0, 'ondisk_log_size': 3064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25056416, 'num_objects': 3059, 'num_object_clones': 0, 'num_object_copies': 12236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3062, 'num_write_kb': 24484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1080}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:03.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:03.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 82 20 50551 50531 4.81355 2.29688 0.0355474 0.0324452 2024-09-10T20:21:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:03.891 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30596/194616 objects degraded (15.721%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":152,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":48654,"data_bytes":398532800,"bytes_used":1249878016,"bytes_avail":385297178624,"bytes_total":386547056640,"degraded_objects":30596,"degraded_total":194616,"degraded_ratio":0.15721215110782258,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":107821,"recovering_keys_per_sec":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"write_bytes_sec":2586350,"read_op_per_sec":0,"write_op_per_sec":315},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:21:03.916 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:21:03.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:04.501 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:04.501 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:04.530 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-10T20:21:03.905377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":402604224,"num_objects":49151,"num_object_clones":0,"num_object_copies":196604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49178,"num_write_kb":393306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49205,"ondisk_log_size":49205,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1271496,"kb_used_data":667720,"kb_used_omap":14,"kb_used_meta":603697,"kb_avail":376215864,"statfs":{"total":386547056640,"available":385245044736,"internally_reserved":0,"allocated":683745280,"data_stored":683041356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":618186294},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,7,11,16,7,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35119104,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.f","version":"32'3027","reported_seq":3068,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.319801+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:20:59.319801+0000","last_peered":"2024-09-10T20:20:59.319801+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:20:59.319801+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3027,"log_dups_size":0,"ondisk_log_size":3027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24797184,"num_objects":3027,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":24216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3084","reported_seq":3146,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.176627+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:20:59.176627+0000","last_peered":"2024-09-10T20:20:59.176627+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:20:59.176627+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3084,"log_dups_size":0,"ondisk_log_size":3084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3082,"num_write_kb":24652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3114","reported_seq":3189,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:02.065517+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:02.065517+0000","last_peered":"2024-09-10T20:21:02.065517+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:02.065517+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3114,"log_dups_size":0,"ondisk_log_size":3114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25483856,"num_objects":3111,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1149}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3111","reported_seq":3173,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.189384+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:20:59.189384+0000","last_peered":"2024-09-10T20:20:59.189384+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:20:59.189384+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25449632,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3109,"num_write_kb":24861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3089","reported_seq":3140,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.333279+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:20:59.333279+0000","last_peered":"2024-09-10T20:20:59.333279+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:20:59.333279+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25277600,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3087,"num_write_kb":24693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3088","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.220247+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:20:59.220247+0000","last_peered":"2024-09-10T20:20:59.220247+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:20:59.220247+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":24704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3076","reported_seq":3139,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.221810+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:20:59.221810+0000","last_peered":"2024-09-10T20:20:59.221810+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:20:59.221810+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3076,"log_dups_size":0,"ondisk_log_size":3076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25146528,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3074,"num_write_kb":24572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1117}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3095","reported_seq":3157,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.147899+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:20:59.147899+0000","last_peered":"2024-09-10T20:20:59.147899+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:20:59.147899+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310368,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3020","reported_seq":4846,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.372747+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:20:59.372747+0000","last_peered":"2024-09-10T20:20:59.372747+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:20:59.372747+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24668480,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3029","reported_seq":3106,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:02.065486+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:02.065486+0000","last_peered":"2024-09-10T20:21:02.065486+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:02.065486+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3029,"log_dups_size":0,"ondisk_log_size":3029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24794272,"num_objects":3027,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":24214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3025","reported_seq":3077,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.376645+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:20:59.376645+0000","last_peered":"2024-09-10T20:20:59.376645+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:20:59.376645+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3025,"log_dups_size":0,"ondisk_log_size":3025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24735472,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3022,"num_write_kb":24164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3093","reported_seq":3136,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.386003+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:20:59.386003+0000","last_peered":"2024-09-10T20:20:59.386003+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:20:59.386003+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25328208,"num_objects":3092,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1114}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3046","reported_seq":3118,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:02.081172+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:02.081172+0000","last_peered":"2024-09-10T20:21:02.081172+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:02.081172+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3046,"log_dups_size":0,"ondisk_log_size":3046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24943184,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3045,"num_write_kb":24359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3068","reported_seq":3118,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.287831+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:20:59.287831+0000","last_peered":"2024-09-10T20:20:59.287831+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:20:59.287831+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25097376,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":24524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3176","reported_seq":3254,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:02.076725+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:02.076725+0000","last_peered":"2024-09-10T20:21:02.076725+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:02.076725+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3176,"log_dups_size":0,"ondisk_log_size":3176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990304,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3174,"num_write_kb":25389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1211}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3064","reported_seq":3127,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:20:59.126137+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:20:59.126137+0000","last_peered":"2024-09-10T20:20:59.126137+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:20:59.126137+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3064,"log_dups_size":0,"ondisk_log_size":3064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3062,"num_write_kb":24484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":402604224,"num_objects":49151,"num_object_clones":0,"num_object_copies":196604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49178,"num_write_kb":393306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":685572096,"data_stored":685572096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49205,"ondisk_log_size":49205,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371316,"kb_used_data":195236,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94000524,"statfs":{"total":96636764160,"available":96256536576,"internally_reserved":0,"allocated":199921664,"data_stored":199745683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379572,"kb_used_data":199844,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93992268,"statfs":{"total":96636764160,"available":96248082432,"internally_reserved":0,"allocated":204640256,"data_stored":204464275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172568,"kb_used_data":78408,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94199272,"statfs":{"total":96636764160,"available":96460054528,"internally_reserved":0,"allocated":80289792,"data_stored":80113811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,11,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348040,"kb_used_data":194232,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94023800,"statfs":{"total":96636764160,"available":96280371200,"internally_reserved":0,"allocated":198893568,"data_stored":198717587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199897088,"data_stored":199897088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80654336,"data_stored":80654336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204832768,"data_stored":204832768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200187904,"data_stored":200187904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:04.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 83 20 50925 50905 4.79076 2.92188 0.0672256 0.0325989 2024-09-10T20:21:05.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 84 20 51338 51318 4.77214 3.22656 0.059327 0.0327292 2024-09-10T20:21:06.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 85 20 51718 51698 4.75092 2.96875 0.0110441 0.0328318 2024-09-10T20:21:07.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:07.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 86 20 52091 52071 4.72956 2.91406 0.0772477 0.0330185 2024-09-10T20:21:08.152 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:08.153 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:08.175 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-10T20:21:07.906304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":418177216,"num_objects":51052,"num_object_clones":0,"num_object_copies":204208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51079,"num_write_kb":408514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":7962624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51106,"ondisk_log_size":51106,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1337936,"kb_used_data":698064,"kb_used_omap":14,"kb_used_meta":639793,"kb_avail":376149424,"statfs":{"total":386547056640,"available":385177010176,"internally_reserved":0,"allocated":714817536,"data_stored":714113612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":655148598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,8,16,24,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34840576,"num_objects":4253,"num_object_clones":0,"num_object_copies":17012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":34024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003298"},"pg_stats":[{"pgid":"1.f","version":"32'3154","reported_seq":3195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.322879+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:04.322879+0000","last_peered":"2024-09-10T20:21:04.322879+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:04.322879+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3154,"log_dups_size":0,"ondisk_log_size":3154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25837568,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3154,"num_write_kb":25232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3201","reported_seq":3262,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.240865+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:04.240865+0000","last_peered":"2024-09-10T20:21:04.240865+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:04.240865+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3201,"log_dups_size":0,"ondisk_log_size":3201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26186912,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3199,"num_write_kb":25588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3216","reported_seq":3291,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.081988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:07.081988+0000","last_peered":"2024-09-10T20:21:07.081988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:07.081988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3216,"log_dups_size":0,"ondisk_log_size":3216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319440,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3257","reported_seq":3318,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.198874+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:04.198874+0000","last_peered":"2024-09-10T20:21:04.198874+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:04.198874+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645664,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3215","reported_seq":3266,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.357775+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:04.357775+0000","last_peered":"2024-09-10T20:21:04.357775+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:04.357775+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26309792,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3214","reported_seq":3255,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.382963+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:04.382963+0000","last_peered":"2024-09-10T20:21:04.382963+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:04.382963+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26329088,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3214,"num_write_kb":25712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3189","reported_seq":3253,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.215596+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:04.215596+0000","last_peered":"2024-09-10T20:21:04.215596+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:04.215596+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26072224,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3187,"num_write_kb":25476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1230}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3209","reported_seq":3271,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.225660+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:04.225660+0000","last_peered":"2024-09-10T20:21:04.225660+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:04.225660+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3209,"log_dups_size":0,"ondisk_log_size":3209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244256,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3207,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3142","reported_seq":5139,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.456570+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:04.456570+0000","last_peered":"2024-09-10T20:21:04.456570+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:04.456570+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25667904,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":25089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":975,"num_bytes_recovered":7962624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3152","reported_seq":3229,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.051518+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:07.051518+0000","last_peered":"2024-09-10T20:21:07.051518+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:07.051518+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3152,"log_dups_size":0,"ondisk_log_size":3152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25801888,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1247}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3139","reported_seq":3192,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.447597+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:04.447597+0000","last_peered":"2024-09-10T20:21:04.447597+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:04.447597+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25669360,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3211","reported_seq":3254,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.410415+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:04.410415+0000","last_peered":"2024-09-10T20:21:04.410415+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:04.410415+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3211,"log_dups_size":0,"ondisk_log_size":3211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26294864,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":25679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3151","reported_seq":3225,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.043328+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:07.043328+0000","last_peered":"2024-09-10T20:21:07.043328+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:07.043328+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25803344,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3195","reported_seq":3245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.408579+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:04.408579+0000","last_peered":"2024-09-10T20:21:04.408579+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:04.408579+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26137760,"num_objects":3191,"num_object_clones":0,"num_object_copies":12764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3298","reported_seq":3376,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.068288+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:07.068288+0000","last_peered":"2024-09-10T20:21:07.068288+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:07.068288+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26989728,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3163","reported_seq":3226,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.184331+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:04.184331+0000","last_peered":"2024-09-10T20:21:04.184331+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:04.184331+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3163,"log_dups_size":0,"ondisk_log_size":3163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25867424,"num_objects":3158,"num_object_clones":0,"num_object_copies":12632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":25276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":418177216,"num_objects":51052,"num_object_clones":0,"num_object_copies":204208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51079,"num_write_kb":408514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":7962624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":716734464,"data_stored":716734464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51106,"ondisk_log_size":51106,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383152,"kb_used_data":202912,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93988688,"statfs":{"total":96636764160,"available":96244416512,"internally_reserved":0,"allocated":207781888,"data_stored":207605907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,5,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391096,"kb_used_data":207208,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93980744,"statfs":{"total":96636764160,"available":96236281856,"internally_reserved":0,"allocated":212180992,"data_stored":212005011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":185648,"kb_used_data":85920,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94186192,"statfs":{"total":96636764160,"available":96446660608,"internally_reserved":0,"allocated":87982080,"data_stored":87806099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":102100690},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378040,"kb_used_data":202024,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93993800,"statfs":{"total":96636764160,"available":96249651200,"internally_reserved":0,"allocated":206872576,"data_stored":206696595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88641536,"data_stored":88641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212262912,"data_stored":212262912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":208093184,"data_stored":208093184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:08.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:08.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 87 20 52458 52438 4.70814 2.86719 0.0621599 0.0331719 2024-09-10T20:21:08.785 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:08.786 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:08.818 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-10T20:21:07.906304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":418177216,"num_objects":51052,"num_object_clones":0,"num_object_copies":204208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51079,"num_write_kb":408514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":7962624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51106,"ondisk_log_size":51106,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1337936,"kb_used_data":698064,"kb_used_omap":14,"kb_used_meta":639793,"kb_avail":376149424,"statfs":{"total":386547056640,"available":385177010176,"internally_reserved":0,"allocated":714817536,"data_stored":714113612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":655148598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,8,16,24,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34840576,"num_objects":4253,"num_object_clones":0,"num_object_copies":17012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":34024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003298"},"pg_stats":[{"pgid":"1.f","version":"32'3154","reported_seq":3195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.322879+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:04.322879+0000","last_peered":"2024-09-10T20:21:04.322879+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:04.322879+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3154,"log_dups_size":0,"ondisk_log_size":3154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25837568,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3154,"num_write_kb":25232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3201","reported_seq":3262,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.240865+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:04.240865+0000","last_peered":"2024-09-10T20:21:04.240865+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:04.240865+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3201,"log_dups_size":0,"ondisk_log_size":3201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26186912,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3199,"num_write_kb":25588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3216","reported_seq":3291,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.081988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:07.081988+0000","last_peered":"2024-09-10T20:21:07.081988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:07.081988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3216,"log_dups_size":0,"ondisk_log_size":3216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319440,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3257","reported_seq":3318,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.198874+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:04.198874+0000","last_peered":"2024-09-10T20:21:04.198874+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:04.198874+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645664,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3215","reported_seq":3266,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.357775+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:04.357775+0000","last_peered":"2024-09-10T20:21:04.357775+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:04.357775+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26309792,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3214","reported_seq":3255,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.382963+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:04.382963+0000","last_peered":"2024-09-10T20:21:04.382963+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:04.382963+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26329088,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3214,"num_write_kb":25712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3189","reported_seq":3253,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.215596+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:04.215596+0000","last_peered":"2024-09-10T20:21:04.215596+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:04.215596+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26072224,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3187,"num_write_kb":25476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1230}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3209","reported_seq":3271,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.225660+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:04.225660+0000","last_peered":"2024-09-10T20:21:04.225660+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:04.225660+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3209,"log_dups_size":0,"ondisk_log_size":3209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244256,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3207,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3142","reported_seq":5139,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.456570+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:04.456570+0000","last_peered":"2024-09-10T20:21:04.456570+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:04.456570+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25667904,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":25089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":975,"num_bytes_recovered":7962624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3152","reported_seq":3229,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.051518+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:07.051518+0000","last_peered":"2024-09-10T20:21:07.051518+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:07.051518+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3152,"log_dups_size":0,"ondisk_log_size":3152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25801888,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1247}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3139","reported_seq":3192,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.447597+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:04.447597+0000","last_peered":"2024-09-10T20:21:04.447597+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:04.447597+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25669360,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3211","reported_seq":3254,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.410415+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:04.410415+0000","last_peered":"2024-09-10T20:21:04.410415+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:04.410415+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3211,"log_dups_size":0,"ondisk_log_size":3211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26294864,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":25679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3151","reported_seq":3225,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.043328+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:07.043328+0000","last_peered":"2024-09-10T20:21:07.043328+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:07.043328+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25803344,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3195","reported_seq":3245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.408579+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:04.408579+0000","last_peered":"2024-09-10T20:21:04.408579+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:04.408579+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26137760,"num_objects":3191,"num_object_clones":0,"num_object_copies":12764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3298","reported_seq":3376,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.068288+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:07.068288+0000","last_peered":"2024-09-10T20:21:07.068288+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:07.068288+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26989728,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3163","reported_seq":3226,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:04.184331+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:04.184331+0000","last_peered":"2024-09-10T20:21:04.184331+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:04.184331+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3163,"log_dups_size":0,"ondisk_log_size":3163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25867424,"num_objects":3158,"num_object_clones":0,"num_object_copies":12632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":25276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":418177216,"num_objects":51052,"num_object_clones":0,"num_object_copies":204208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51079,"num_write_kb":408514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":7962624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":716734464,"data_stored":716734464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51106,"ondisk_log_size":51106,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383152,"kb_used_data":202912,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93988688,"statfs":{"total":96636764160,"available":96244416512,"internally_reserved":0,"allocated":207781888,"data_stored":207605907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,5,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391096,"kb_used_data":207208,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93980744,"statfs":{"total":96636764160,"available":96236281856,"internally_reserved":0,"allocated":212180992,"data_stored":212005011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":185648,"kb_used_data":85920,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94186192,"statfs":{"total":96636764160,"available":96446660608,"internally_reserved":0,"allocated":87982080,"data_stored":87806099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":102100690},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378040,"kb_used_data":202024,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93993800,"statfs":{"total":96636764160,"available":96249651200,"internally_reserved":0,"allocated":206872576,"data_stored":206696595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88641536,"data_stored":88641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212262912,"data_stored":212262912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":208093184,"data_stored":208093184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:08.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:08.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3154", 'reported_seq': 3195, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.322879+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:04.322879+0000', 'last_peered': '2024-09-10T20:21:04.322879+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:04.322879+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3154, 'log_dups_size': 0, 'ondisk_log_size': 3154, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25837568, 'num_objects': 3154, 'num_object_clones': 0, 'num_object_copies': 12616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3154, 'num_write_kb': 25232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1188}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:08.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:21:08.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3201", 'reported_seq': 3262, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.240865+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:21:04.240865+0000', 'last_peered': '2024-09-10T20:21:04.240865+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:21:04.240865+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3201, 'log_dups_size': 0, 'ondisk_log_size': 3201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26186912, 'num_objects': 3197, 'num_object_clones': 0, 'num_object_copies': 12788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3199, 'num_write_kb': 25588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:08.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:08.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3216", 'reported_seq': 3291, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:07.081988+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:07.081988+0000', 'last_peered': '2024-09-10T20:21:07.081988+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:07.081988+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3216, 'log_dups_size': 0, 'ondisk_log_size': 3216, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26319440, 'num_objects': 3213, 'num_object_clones': 0, 'num_object_copies': 12852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3215, 'num_write_kb': 25710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1251}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:08.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:08.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3257", 'reported_seq': 3318, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.198874+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:21:04.198874+0000', 'last_peered': '2024-09-10T20:21:04.198874+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:04.198874+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3257, 'log_dups_size': 0, 'ondisk_log_size': 3257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26645664, 'num_objects': 3253, 'num_object_clones': 0, 'num_object_copies': 13012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3255, 'num_write_kb': 26029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3215", 'reported_seq': 3266, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.357775+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:04.357775+0000', 'last_peered': '2024-09-10T20:21:04.357775+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:04.357775+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3215, 'log_dups_size': 0, 'ondisk_log_size': 3215, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26309792, 'num_objects': 3212, 'num_object_clones': 0, 'num_object_copies': 12848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3213, 'num_write_kb': 25701, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1232}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3214", 'reported_seq': 3255, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.382963+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:04.382963+0000', 'last_peered': '2024-09-10T20:21:04.382963+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:04.382963+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3214, 'log_dups_size': 0, 'ondisk_log_size': 3214, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26329088, 'num_objects': 3214, 'num_object_clones': 0, 'num_object_copies': 12856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3214, 'num_write_kb': 25712, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1276}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3189", 'reported_seq': 3253, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.215596+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:04.215596+0000', 'last_peered': '2024-09-10T20:21:04.215596+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:04.215596+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3189, 'log_dups_size': 0, 'ondisk_log_size': 3189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26072224, 'num_objects': 3183, 'num_object_clones': 0, 'num_object_copies': 12732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3187, 'num_write_kb': 25476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1230}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3209", 'reported_seq': 3271, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.225660+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:04.225660+0000', 'last_peered': '2024-09-10T20:21:04.225660+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:04.225660+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3209, 'log_dups_size': 0, 'ondisk_log_size': 3209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26244256, 'num_objects': 3204, 'num_object_clones': 0, 'num_object_copies': 12816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3207, 'num_write_kb': 25644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3142", 'reported_seq': 5139, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.456570+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:21:04.456570+0000', 'last_peered': '2024-09-10T20:21:04.456570+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:21:04.456570+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3142, 'log_dups_size': 0, 'ondisk_log_size': 3142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25667904, 'num_objects': 3134, 'num_object_clones': 0, 'num_object_copies': 12536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 948, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3139, 'num_write_kb': 25089, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 975, 'num_bytes_recovered': 7962624, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:08.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3152", 'reported_seq': 3229, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:07.051518+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:07.051518+0000', 'last_peered': '2024-09-10T20:21:07.051518+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:07.051518+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3152, 'log_dups_size': 0, 'ondisk_log_size': 3152, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25801888, 'num_objects': 3150, 'num_object_clones': 0, 'num_object_copies': 12600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3150, 'num_write_kb': 25198, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1247}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3139", 'reported_seq': 3192, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.447597+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:04.447597+0000', 'last_peered': '2024-09-10T20:21:04.447597+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:04.447597+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3139, 'log_dups_size': 0, 'ondisk_log_size': 3139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25669360, 'num_objects': 3134, 'num_object_clones': 0, 'num_object_copies': 12536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3136, 'num_write_kb': 25076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3211", 'reported_seq': 3254, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.410415+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:04.410415+0000', 'last_peered': '2024-09-10T20:21:04.410415+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:04.410415+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3211, 'log_dups_size': 0, 'ondisk_log_size': 3211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26294864, 'num_objects': 3210, 'num_object_clones': 0, 'num_object_copies': 12840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3210, 'num_write_kb': 25679, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3151", 'reported_seq': 3225, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:07.043328+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:07.043328+0000', 'last_peered': '2024-09-10T20:21:07.043328+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:07.043328+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3151, 'log_dups_size': 0, 'ondisk_log_size': 3151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25803344, 'num_objects': 3150, 'num_object_clones': 0, 'num_object_copies': 12600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3150, 'num_write_kb': 25199, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3195", 'reported_seq': 3245, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.408579+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:04.408579+0000', 'last_peered': '2024-09-10T20:21:04.408579+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:04.408579+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3195, 'log_dups_size': 0, 'ondisk_log_size': 3195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26137760, 'num_objects': 3191, 'num_object_clones': 0, 'num_object_copies': 12764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3193, 'num_write_kb': 25540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:08.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3298", 'reported_seq': 3376, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:07.068288+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:07.068288+0000', 'last_peered': '2024-09-10T20:21:07.068288+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:07.068288+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3298, 'log_dups_size': 0, 'ondisk_log_size': 3298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26989728, 'num_objects': 3295, 'num_object_clones': 0, 'num_object_copies': 13180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3296, 'num_write_kb': 26365, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:08.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:08.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3163", 'reported_seq': 3226, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:04.184331+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:04.184331+0000', 'last_peered': '2024-09-10T20:21:04.184331+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:04.184331+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3163, 'log_dups_size': 0, 'ondisk_log_size': 3163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25867424, 'num_objects': 3158, 'num_object_clones': 0, 'num_object_copies': 12632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3161, 'num_write_kb': 25276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:08.823 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:09.598 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:09.598 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30509/204208 objects degraded (14.940%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":158,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51052,"data_bytes":418177216,"bytes_used":1370046464,"bytes_avail":385177010176,"bytes_total":386547056640,"degraded_objects":30509,"degraded_total":204208,"degraded_ratio":0.14940159053514065,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":116021,"recovering_keys_per_sec":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"write_bytes_sec":2902583,"read_op_per_sec":0,"write_op_per_sec":354},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:20:29.893387+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:21:09.624 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:21:09.625 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:09.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 88 20 52820 52800 4.68678 2.82812 0.0517776 0.0333253 2024-09-10T20:21:10.194 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:10.194 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:10.222 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-10T20:21:09.907039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429932736,"num_objects":52487,"num_object_clones":0,"num_object_copies":209948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52514,"num_write_kb":419994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":8536064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52541,"ondisk_log_size":52541,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1365508,"kb_used_data":713156,"kb_used_omap":14,"kb_used_meta":652273,"kb_avail":376121852,"statfs":{"total":386547056640,"available":385148776448,"internally_reserved":0,"allocated":730271744,"data_stored":729567820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":667928118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,11,13,25,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42901504,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003543"},"pg_stats":[{"pgid":"1.f","version":"32'3291","reported_seq":3332,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.469573+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:09.469573+0000","last_peered":"2024-09-10T20:21:09.469573+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:09.469573+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3291,"log_dups_size":0,"ondisk_log_size":3291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26959872,"num_objects":3291,"num_object_clones":0,"num_object_copies":13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3291,"num_write_kb":26328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3325","reported_seq":3388,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.170223+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:09.170223+0000","last_peered":"2024-09-10T20:21:09.170223+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:09.170223+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202720,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3323,"num_write_kb":26580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3216","reported_seq":3291,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.081988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:07.081988+0000","last_peered":"2024-09-10T20:21:07.081988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:07.081988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3216,"log_dups_size":0,"ondisk_log_size":3216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319440,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3371","reported_seq":3433,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.235505+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:09.235505+0000","last_peered":"2024-09-10T20:21:09.235505+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:09.235505+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3371,"log_dups_size":0,"ondisk_log_size":3371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27579552,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3369,"num_write_kb":26941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3334","reported_seq":3385,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.478884+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:09.478884+0000","last_peered":"2024-09-10T20:21:09.478884+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:09.478884+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27284640,"num_objects":3331,"num_object_clones":0,"num_object_copies":13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":26653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3326","reported_seq":3367,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.443166+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:09.443166+0000","last_peered":"2024-09-10T20:21:09.443166+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:09.443166+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27246592,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":26608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3309","reported_seq":3373,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.203457+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:09.203457+0000","last_peered":"2024-09-10T20:21:09.203457+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:09.203457+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27055264,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":26436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1350}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3334","reported_seq":3397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.169482+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:09.169482+0000","last_peered":"2024-09-10T20:21:09.169482+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:09.169482+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27268256,"num_objects":3329,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":26644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3264","reported_seq":5410,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.457929+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:09.457929+0000","last_peered":"2024-09-10T20:21:09.457929+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:09.457929+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26667328,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3261,"num_write_kb":26065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1048,"num_bytes_recovered":8536064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3152","reported_seq":3229,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.051518+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:07.051518+0000","last_peered":"2024-09-10T20:21:07.051518+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:07.051518+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3152,"log_dups_size":0,"ondisk_log_size":3152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25801888,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1247}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3265","reported_seq":3319,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.204644+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:09.204644+0000","last_peered":"2024-09-10T20:21:09.204644+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:09.204644+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26701552,"num_objects":3260,"num_object_clones":0,"num_object_copies":13040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3314","reported_seq":3357,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.284973+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:09.284973+0000","last_peered":"2024-09-10T20:21:09.284973+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:09.284973+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138640,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3151","reported_seq":3225,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.043328+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:07.043328+0000","last_peered":"2024-09-10T20:21:07.043328+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:07.043328+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25803344,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3307","reported_seq":3357,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.374735+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:09.374735+0000","last_peered":"2024-09-10T20:21:09.374735+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:09.374735+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27055264,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3305,"num_write_kb":26436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1328}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3298","reported_seq":3376,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:07.068288+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:07.068288+0000","last_peered":"2024-09-10T20:21:07.068288+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:07.068288+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26989728,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3284","reported_seq":3347,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.044253+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:09.044253+0000","last_peered":"2024-09-10T20:21:09.044253+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:09.044253+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3284,"log_dups_size":0,"ondisk_log_size":3284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858656,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3282,"num_write_kb":26244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":429932736,"num_objects":52487,"num_object_clones":0,"num_object_copies":209948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52514,"num_write_kb":419994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":8536064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":732409856,"data_stored":732409856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52541,"ondisk_log_size":52541,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395120,"kb_used_data":210720,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93976720,"statfs":{"total":96636764160,"available":96232161280,"internally_reserved":0,"allocated":215777280,"data_stored":215601299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,5,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391096,"kb_used_data":207208,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93980744,"statfs":{"total":96636764160,"available":96236281856,"internally_reserved":0,"allocated":212180992,"data_stored":212005011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":185648,"kb_used_data":85920,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94186192,"statfs":{"total":96636764160,"available":96446660608,"internally_reserved":0,"allocated":87982080,"data_stored":87806099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":102100690},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393644,"kb_used_data":209308,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93978196,"statfs":{"total":96636764160,"available":96233672704,"internally_reserved":0,"allocated":214331392,"data_stored":214155411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188739279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215564288,"data_stored":215564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88641536,"data_stored":88641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212262912,"data_stored":212262912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215941120,"data_stored":215941120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:10.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 89 20 53199 53179 4.66738 2.96094 0.0312512 0.0334604 2024-09-10T20:21:11.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 90 20 53612 53592 4.65137 3.22656 0.0596151 0.0335786 2024-09-10T20:21:12.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 91 20 53963 53943 4.63039 2.74219 0.0556335 0.0337324 2024-09-10T20:21:13.224 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:13.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 92 20 54341 54321 4.61216 2.95312 0.0408386 0.0338647 2024-09-10T20:21:14.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 93 20 54749 54729 4.59684 3.1875 0.0527015 0.0339767 2024-09-10T20:21:15.474 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:15.475 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:15.497 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-10T20:21:13.910529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433651904,"num_objects":52941,"num_object_clones":0,"num_object_copies":211764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52968,"num_write_kb":423626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":8536064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52995,"ondisk_log_size":52995,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1390520,"kb_used_data":728440,"kb_used_omap":14,"kb_used_meta":662001,"kb_avail":376096840,"statfs":{"total":386547056640,"available":385123164160,"internally_reserved":0,"allocated":745922560,"data_stored":745218636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":677889590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,4,14,10,14,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35119104,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005560"},"pg_stats":[{"pgid":"1.f","version":"32'3291","reported_seq":3332,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.469573+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:09.469573+0000","last_peered":"2024-09-10T20:21:09.469573+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:09.469573+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3291,"log_dups_size":0,"ondisk_log_size":3291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26959872,"num_objects":3291,"num_object_clones":0,"num_object_copies":13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3291,"num_write_kb":26328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3325","reported_seq":3388,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.170223+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:09.170223+0000","last_peered":"2024-09-10T20:21:09.170223+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:09.170223+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202720,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3323,"num_write_kb":26580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3331","reported_seq":3406,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.877196+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:11.877196+0000","last_peered":"2024-09-10T20:21:11.877196+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:11.877196+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3331,"log_dups_size":0,"ondisk_log_size":3331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27261520,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3330,"num_write_kb":26630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1366}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3371","reported_seq":3433,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.235505+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:09.235505+0000","last_peered":"2024-09-10T20:21:09.235505+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:09.235505+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3371,"log_dups_size":0,"ondisk_log_size":3371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27579552,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3369,"num_write_kb":26941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3334","reported_seq":3385,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.478884+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:09.478884+0000","last_peered":"2024-09-10T20:21:09.478884+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:09.478884+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27284640,"num_objects":3331,"num_object_clones":0,"num_object_copies":13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":26653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3326","reported_seq":3367,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.443166+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:09.443166+0000","last_peered":"2024-09-10T20:21:09.443166+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:09.443166+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27246592,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":26608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3309","reported_seq":3373,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.203457+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:09.203457+0000","last_peered":"2024-09-10T20:21:09.203457+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:09.203457+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27055264,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":26436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1350}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3334","reported_seq":3397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.169482+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:09.169482+0000","last_peered":"2024-09-10T20:21:09.169482+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:09.169482+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27268256,"num_objects":3329,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":26644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3264","reported_seq":5410,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.457929+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:09.457929+0000","last_peered":"2024-09-10T20:21:09.457929+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:09.457929+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26667328,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3261,"num_write_kb":26065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1048,"num_bytes_recovered":8536064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3257","reported_seq":3334,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.858170+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:11.858170+0000","last_peered":"2024-09-10T20:21:11.858170+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:11.858170+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26662048,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3265","reported_seq":3319,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.204644+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:09.204644+0000","last_peered":"2024-09-10T20:21:09.204644+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:09.204644+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26701552,"num_objects":3260,"num_object_clones":0,"num_object_copies":13040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3314","reported_seq":3357,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.284973+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:09.284973+0000","last_peered":"2024-09-10T20:21:09.284973+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:09.284973+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138640,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3267","reported_seq":3341,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.869823+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:11.869823+0000","last_peered":"2024-09-10T20:21:11.869823+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:11.869823+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26753616,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":26127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3307","reported_seq":3357,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.374735+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:09.374735+0000","last_peered":"2024-09-10T20:21:09.374735+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:09.374735+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27055264,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3305,"num_write_kb":26436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1328}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3416","reported_seq":3494,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.875410+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:11.875410+0000","last_peered":"2024-09-10T20:21:11.875410+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:11.875410+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3416,"log_dups_size":0,"ondisk_log_size":3416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956384,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3414,"num_write_kb":27309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3284","reported_seq":3347,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:09.044253+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:09.044253+0000","last_peered":"2024-09-10T20:21:09.044253+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:09.044253+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3284,"log_dups_size":0,"ondisk_log_size":3284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858656,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3282,"num_write_kb":26244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":433651904,"num_objects":52941,"num_object_clones":0,"num_object_copies":211764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52968,"num_write_kb":423626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":8536064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":747991040,"data_stored":747991040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52995,"ondisk_log_size":52995,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395120,"kb_used_data":210720,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93976720,"statfs":{"total":96636764160,"available":96232161280,"internally_reserved":0,"allocated":215777280,"data_stored":215601299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,5,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386280,"kb_used_data":214872,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93985560,"statfs":{"total":96636764160,"available":96241213440,"internally_reserved":0,"allocated":220028928,"data_stored":219852947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215476,"kb_used_data":93540,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94156364,"statfs":{"total":96636764160,"available":96416116736,"internally_reserved":0,"allocated":95784960,"data_stored":95608979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393644,"kb_used_data":209308,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93978196,"statfs":{"total":96636764160,"available":96233672704,"internally_reserved":0,"allocated":214331392,"data_stored":214155411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188739279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215564288,"data_stored":215564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96608256,"data_stored":96608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219877376,"data_stored":219877376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215941120,"data_stored":215941120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:15.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:15.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 94 20 55146 55126 4.58093 3.10156 0.0380451 0.0340933 2024-09-10T20:21:16.111 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:16.140 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-10T20:21:15.910985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":445399232,"num_objects":54375,"num_object_clones":0,"num_object_copies":217500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":435098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1155,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54429,"ondisk_log_size":54429,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1382156,"kb_used_data":745036,"kb_used_omap":14,"kb_used_meta":637041,"kb_avail":376105204,"statfs":{"total":386547056640,"available":385131728896,"internally_reserved":0,"allocated":762916864,"data_stored":762212940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":652330550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,5,8,7,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42795008,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005608"},"pg_stats":[{"pgid":"1.f","version":"32'3411","reported_seq":3449,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.480315+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:14.480315+0000","last_peered":"2024-09-10T20:21:14.480315+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:14.480315+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27942912,"num_objects":3411,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3427","reported_seq":3490,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.176981+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:14.176981+0000","last_peered":"2024-09-10T20:21:14.176981+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:14.176981+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28038304,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3425,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3331","reported_seq":3406,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.877196+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:11.877196+0000","last_peered":"2024-09-10T20:21:11.877196+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:11.877196+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3331,"log_dups_size":0,"ondisk_log_size":3331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27261520,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3330,"num_write_kb":26630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1366}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3505","reported_seq":3566,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.210301+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:14.210301+0000","last_peered":"2024-09-10T20:21:14.210301+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:14.210301+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677280,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":28013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3443","reported_seq":3494,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.314752+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:14.314752+0000","last_peered":"2024-09-10T20:21:14.314752+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:14.314752+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177568,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3441,"num_write_kb":27525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3445","reported_seq":3486,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.405380+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:14.405380+0000","last_peered":"2024-09-10T20:21:14.405380+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:14.405380+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3445,"log_dups_size":0,"ondisk_log_size":3445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28221440,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3445,"num_write_kb":27560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3420","reported_seq":3484,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.172192+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:14.172192+0000","last_peered":"2024-09-10T20:21:14.172192+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:14.172192+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3420,"log_dups_size":0,"ondisk_log_size":3420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27964576,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":27324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1461}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3479","reported_seq":3542,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.198156+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:14.198156+0000","last_peered":"2024-09-10T20:21:14.198156+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:14.198156+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":27804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3364","reported_seq":5676,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.523867+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:14.523867+0000","last_peered":"2024-09-10T20:21:14.523867+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:14.523867+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3364,"log_dups_size":0,"ondisk_log_size":3364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27486528,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":26865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1131,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3257","reported_seq":3334,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.858170+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:11.858170+0000","last_peered":"2024-09-10T20:21:11.858170+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:11.858170+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26662048,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3393","reported_seq":3447,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.484081+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:14.484081+0000","last_peered":"2024-09-10T20:21:14.484081+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:14.484081+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27750128,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3390,"num_write_kb":27108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3449","reported_seq":3492,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.476857+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:14.476857+0000","last_peered":"2024-09-10T20:21:14.476857+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:14.476857+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28244560,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1470}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3267","reported_seq":3341,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.869823+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:11.869823+0000","last_peered":"2024-09-10T20:21:11.869823+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:11.869823+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26753616,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":26127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3429","reported_seq":3479,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.446993+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:14.446993+0000","last_peered":"2024-09-10T20:21:14.446993+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:14.446993+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28054688,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3427,"num_write_kb":27412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3416","reported_seq":3494,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.875410+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:11.875410+0000","last_peered":"2024-09-10T20:21:11.875410+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:11.875410+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3416,"log_dups_size":0,"ondisk_log_size":3416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956384,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3414,"num_write_kb":27309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3393","reported_seq":3456,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.200816+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:14.200816+0000","last_peered":"2024-09-10T20:21:14.200816+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:14.200816+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27751584,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":445399232,"num_objects":54375,"num_object_clones":0,"num_object_copies":217500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":435098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1155,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":763654144,"data_stored":763654144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54429,"ondisk_log_size":54429,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390168,"kb_used_data":218248,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93981672,"statfs":{"total":96636764160,"available":96237232128,"internally_reserved":0,"allocated":223485952,"data_stored":223309971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386280,"kb_used_data":214872,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93985560,"statfs":{"total":96636764160,"available":96241213440,"internally_reserved":0,"allocated":220028928,"data_stored":219852947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215476,"kb_used_data":93540,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94156364,"statfs":{"total":96636764160,"available":96416116736,"internally_reserved":0,"allocated":95784960,"data_stored":95608979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390232,"kb_used_data":218376,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93981608,"statfs":{"total":96636764160,"available":96237166592,"internally_reserved":0,"allocated":223617024,"data_stored":223441043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":175959759},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":223350784,"data_stored":223350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96608256,"data_stored":96608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219877376,"data_stored":219877376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223817728,"data_stored":223817728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:16.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3411", 'reported_seq': 3449, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.480315+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:14.480315+0000', 'last_peered': '2024-09-10T20:21:14.480315+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:14.480315+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3411, 'log_dups_size': 0, 'ondisk_log_size': 3411, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27942912, 'num_objects': 3411, 'num_object_clones': 0, 'num_object_copies': 13644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3411, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3411, 'num_write_kb': 27288, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1445}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3427", 'reported_seq': 3490, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.176981+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:21:14.176981+0000', 'last_peered': '2024-09-10T20:21:14.176981+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:21:14.176981+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3427, 'log_dups_size': 0, 'ondisk_log_size': 3427, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28038304, 'num_objects': 3423, 'num_object_clones': 0, 'num_object_copies': 13692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3425, 'num_write_kb': 27396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1432}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3331", 'reported_seq': 3406, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:11.877196+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:11.877196+0000', 'last_peered': '2024-09-10T20:21:11.877196+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:11.877196+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3331, 'log_dups_size': 0, 'ondisk_log_size': 3331, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27261520, 'num_objects': 3328, 'num_object_clones': 0, 'num_object_copies': 13312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3330, 'num_write_kb': 26630, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1366}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3505", 'reported_seq': 3566, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.210301+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:21:14.210301+0000', 'last_peered': '2024-09-10T20:21:14.210301+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:14.210301+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3505, 'log_dups_size': 0, 'ondisk_log_size': 3505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28677280, 'num_objects': 3501, 'num_object_clones': 0, 'num_object_copies': 14004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3503, 'num_write_kb': 28013, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:16.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3443", 'reported_seq': 3494, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.314752+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:14.314752+0000', 'last_peered': '2024-09-10T20:21:14.314752+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:14.314752+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3443, 'log_dups_size': 0, 'ondisk_log_size': 3443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28177568, 'num_objects': 3440, 'num_object_clones': 0, 'num_object_copies': 13760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3441, 'num_write_kb': 27525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1460}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3445", 'reported_seq': 3486, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.405380+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:14.405380+0000', 'last_peered': '2024-09-10T20:21:14.405380+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:14.405380+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3445, 'log_dups_size': 0, 'ondisk_log_size': 3445, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28221440, 'num_objects': 3445, 'num_object_clones': 0, 'num_object_copies': 13780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3445, 'num_write_kb': 27560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3420", 'reported_seq': 3484, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.172192+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:14.172192+0000', 'last_peered': '2024-09-10T20:21:14.172192+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:14.172192+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3420, 'log_dups_size': 0, 'ondisk_log_size': 3420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27964576, 'num_objects': 3414, 'num_object_clones': 0, 'num_object_copies': 13656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3418, 'num_write_kb': 27324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1461}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3479", 'reported_seq': 3542, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.198156+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:14.198156+0000', 'last_peered': '2024-09-10T20:21:14.198156+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:14.198156+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3479, 'log_dups_size': 0, 'ondisk_log_size': 3479, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28456096, 'num_objects': 3474, 'num_object_clones': 0, 'num_object_copies': 13896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3477, 'num_write_kb': 27804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3364", 'reported_seq': 5676, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.523867+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:21:14.523867+0000', 'last_peered': '2024-09-10T20:21:14.523867+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:21:14.523867+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3364, 'log_dups_size': 0, 'ondisk_log_size': 3364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27486528, 'num_objects': 3356, 'num_object_clones': 0, 'num_object_copies': 13424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 795, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3361, 'num_write_kb': 26865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1131, 'num_bytes_recovered': 9216000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2561}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:16.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3257", 'reported_seq': 3334, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:11.858170+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:11.858170+0000', 'last_peered': '2024-09-10T20:21:11.858170+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:11.858170+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3257, 'log_dups_size': 0, 'ondisk_log_size': 3257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26662048, 'num_objects': 3255, 'num_object_clones': 0, 'num_object_copies': 13020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3255, 'num_write_kb': 26038, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1352}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3393", 'reported_seq': 3447, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.484081+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:14.484081+0000', 'last_peered': '2024-09-10T20:21:14.484081+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:14.484081+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3393, 'log_dups_size': 0, 'ondisk_log_size': 3393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27750128, 'num_objects': 3388, 'num_object_clones': 0, 'num_object_copies': 13552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3390, 'num_write_kb': 27108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1432}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3449", 'reported_seq': 3492, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.476857+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:14.476857+0000', 'last_peered': '2024-09-10T20:21:14.476857+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:14.476857+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3449, 'log_dups_size': 0, 'ondisk_log_size': 3449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28244560, 'num_objects': 3448, 'num_object_clones': 0, 'num_object_copies': 13792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3448, 'num_write_kb': 27583, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1470}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3267", 'reported_seq': 3341, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:11.869823+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:11.869823+0000', 'last_peered': '2024-09-10T20:21:11.869823+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:11.869823+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3267, 'log_dups_size': 0, 'ondisk_log_size': 3267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26753616, 'num_objects': 3266, 'num_object_clones': 0, 'num_object_copies': 13064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3266, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3266, 'num_write_kb': 26127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:16.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3429", 'reported_seq': 3479, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.446993+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:14.446993+0000', 'last_peered': '2024-09-10T20:21:14.446993+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:14.446993+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3429, 'log_dups_size': 0, 'ondisk_log_size': 3429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28054688, 'num_objects': 3425, 'num_object_clones': 0, 'num_object_copies': 13700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3427, 'num_write_kb': 27412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:16.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:16.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3416", 'reported_seq': 3494, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:11.875410+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:11.875410+0000', 'last_peered': '2024-09-10T20:21:11.875410+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:11.875410+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3416, 'log_dups_size': 0, 'ondisk_log_size': 3416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27956384, 'num_objects': 3413, 'num_object_clones': 0, 'num_object_copies': 13652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3414, 'num_write_kb': 27309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:16.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:16.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3393", 'reported_seq': 3456, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:14.200816+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:14.200816+0000', 'last_peered': '2024-09-10T20:21:14.200816+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:14.200816+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3393, 'log_dups_size': 0, 'ondisk_log_size': 3393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27751584, 'num_objects': 3388, 'num_object_clones': 0, 'num_object_copies': 13552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3391, 'num_write_kb': 27116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:16.145 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:16.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 95 20 55506 55486 4.56231 2.8125 0.0723418 0.0342339 2024-09-10T20:21:16.875 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:16.875 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30356/217500 objects degraded (13.957%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":165,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54375,"data_bytes":445399232,"bytes_used":1415327744,"bytes_avail":385131728896,"bytes_total":386547056640,"degraded_objects":30356,"degraded_total":217500,"degraded_ratio":0.13956781609195401,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":163763,"recovering_keys_per_sec":0,"num_objects_recovered":243,"num_bytes_recovered":1966080,"num_keys_recovered":0,"write_bytes_sec":3564584,"read_op_per_sec":0,"write_op_per_sec":435},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-10T20:21:13.910600+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:21:16.912 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:21:16.912 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:17.517 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:17.517 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:17.541 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-10T20:21:15.910985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":445399232,"num_objects":54375,"num_object_clones":0,"num_object_copies":217500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":435098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1155,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54429,"ondisk_log_size":54429,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1382156,"kb_used_data":745036,"kb_used_omap":14,"kb_used_meta":637041,"kb_avail":376105204,"statfs":{"total":386547056640,"available":385131728896,"internally_reserved":0,"allocated":762916864,"data_stored":762212940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":652330550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,5,8,7,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42795008,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005608"},"pg_stats":[{"pgid":"1.f","version":"32'3411","reported_seq":3449,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.480315+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:14.480315+0000","last_peered":"2024-09-10T20:21:14.480315+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:14.480315+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27942912,"num_objects":3411,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3427","reported_seq":3490,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.176981+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:14.176981+0000","last_peered":"2024-09-10T20:21:14.176981+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:14.176981+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28038304,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3425,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3331","reported_seq":3406,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.877196+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:11.877196+0000","last_peered":"2024-09-10T20:21:11.877196+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:11.877196+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3331,"log_dups_size":0,"ondisk_log_size":3331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27261520,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3330,"num_write_kb":26630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1366}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3505","reported_seq":3566,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.210301+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:14.210301+0000","last_peered":"2024-09-10T20:21:14.210301+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:14.210301+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677280,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":28013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3443","reported_seq":3494,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.314752+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:14.314752+0000","last_peered":"2024-09-10T20:21:14.314752+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:14.314752+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177568,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3441,"num_write_kb":27525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3445","reported_seq":3486,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.405380+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:14.405380+0000","last_peered":"2024-09-10T20:21:14.405380+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:14.405380+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3445,"log_dups_size":0,"ondisk_log_size":3445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28221440,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3445,"num_write_kb":27560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3420","reported_seq":3484,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.172192+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:14.172192+0000","last_peered":"2024-09-10T20:21:14.172192+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:14.172192+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3420,"log_dups_size":0,"ondisk_log_size":3420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27964576,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":27324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1461}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3479","reported_seq":3542,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.198156+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:14.198156+0000","last_peered":"2024-09-10T20:21:14.198156+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:14.198156+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":27804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3364","reported_seq":5676,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.523867+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:14.523867+0000","last_peered":"2024-09-10T20:21:14.523867+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:14.523867+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3364,"log_dups_size":0,"ondisk_log_size":3364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27486528,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":26865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1131,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3257","reported_seq":3334,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.858170+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:11.858170+0000","last_peered":"2024-09-10T20:21:11.858170+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:11.858170+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26662048,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3393","reported_seq":3447,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.484081+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:14.484081+0000","last_peered":"2024-09-10T20:21:14.484081+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:14.484081+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27750128,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3390,"num_write_kb":27108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3449","reported_seq":3492,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.476857+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:14.476857+0000","last_peered":"2024-09-10T20:21:14.476857+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:14.476857+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28244560,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1470}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3267","reported_seq":3341,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.869823+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:11.869823+0000","last_peered":"2024-09-10T20:21:11.869823+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:11.869823+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26753616,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":26127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3429","reported_seq":3479,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.446993+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:14.446993+0000","last_peered":"2024-09-10T20:21:14.446993+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:14.446993+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28054688,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3427,"num_write_kb":27412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3416","reported_seq":3494,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:11.875410+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:11.875410+0000","last_peered":"2024-09-10T20:21:11.875410+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:11.875410+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3416,"log_dups_size":0,"ondisk_log_size":3416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956384,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3414,"num_write_kb":27309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3393","reported_seq":3456,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:14.200816+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:14.200816+0000","last_peered":"2024-09-10T20:21:14.200816+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:14.200816+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27751584,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":445399232,"num_objects":54375,"num_object_clones":0,"num_object_copies":217500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":435098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1155,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":763654144,"data_stored":763654144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54429,"ondisk_log_size":54429,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390168,"kb_used_data":218248,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93981672,"statfs":{"total":96636764160,"available":96237232128,"internally_reserved":0,"allocated":223485952,"data_stored":223309971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386280,"kb_used_data":214872,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93985560,"statfs":{"total":96636764160,"available":96241213440,"internally_reserved":0,"allocated":220028928,"data_stored":219852947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215476,"kb_used_data":93540,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94156364,"statfs":{"total":96636764160,"available":96416116736,"internally_reserved":0,"allocated":95784960,"data_stored":95608979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390232,"kb_used_data":218376,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93981608,"statfs":{"total":96636764160,"available":96237166592,"internally_reserved":0,"allocated":223617024,"data_stored":223441043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":175959759},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":223350784,"data_stored":223350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96608256,"data_stored":96608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219877376,"data_stored":219877376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223817728,"data_stored":223817728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:17.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 96 20 55867 55847 4.54417 2.82031 0.115325 0.03437 2024-09-10T20:21:18.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 97 20 55960 55940 4.50481 0.726562 0.00666907 0.0343759 2024-09-10T20:21:19.783 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 98 20 55960 55940 4.45885 0 - 0.0343759 2024-09-10T20:21:20.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:20.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 99 20 55960 55940 4.41381 0 - 0.0343759 2024-09-10T20:21:21.655 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:21.656 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:21.669 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-10T20:21:21.140311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457916608,"num_objects":55903,"num_object_clones":0,"num_object_copies":223612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55930,"num_write_kb":447322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55957,"ondisk_log_size":55957,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1426148,"kb_used_data":772388,"kb_used_omap":14,"kb_used_meta":653681,"kb_avail":376061212,"statfs":{"total":386547056640,"available":385086681088,"internally_reserved":0,"allocated":790925312,"data_stored":790221388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":669369910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,7,7,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27983872,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":27328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.233271"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3523,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.955989+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:17.955989+0000","last_peered":"2024-09-10T20:21:17.955989+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:17.955989+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":3580,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.979163+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:17.979163+0000","last_peered":"2024-09-10T20:21:17.979163+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:17.979163+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3449","reported_seq":3525,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.013892+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:17.013892+0000","last_peered":"2024-09-10T20:21:17.013892+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:17.013892+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28228176,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3663,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.905169+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:17.905169+0000","last_peered":"2024-09-10T20:21:17.905169+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:17.905169+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3581,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.969381+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:17.969381+0000","last_peered":"2024-09-10T20:21:17.969381+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:17.969381+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3572,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:18.027497+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:18.027497+0000","last_peered":"2024-09-10T20:21:18.027497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:18.027497+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3574,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.962718+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:17.962718+0000","last_peered":"2024-09-10T20:21:17.962718+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:17.962718+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3616,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.974792+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:17.974792+0000","last_peered":"2024-09-10T20:21:17.974792+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:17.974792+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":6781,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:19.531780+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:19.531780+0000","last_peered":"2024-09-10T20:21:19.531780+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:19.531780+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28166464,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3444,"num_write_kb":27529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1642,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3383","reported_seq":3460,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:16.998448+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:16.998448+0000","last_peered":"2024-09-10T20:21:16.998448+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:16.998448+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3383,"log_dups_size":0,"ondisk_log_size":3383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694240,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3381,"num_write_kb":27046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3528,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.957343+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:17.957343+0000","last_peered":"2024-09-10T20:21:17.957343+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:17.957343+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.912389+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:17.912389+0000","last_peered":"2024-09-10T20:21:17.912389+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:17.912389+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3404","reported_seq":3478,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.002218+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:17.002218+0000","last_peered":"2024-09-10T20:21:17.002218+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:17.002218+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3404,"log_dups_size":0,"ondisk_log_size":3404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27875920,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3403,"num_write_kb":27223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3579,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.937413+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:17.937413+0000","last_peered":"2024-09-10T20:21:17.937413+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:17.937413+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3538","reported_seq":3616,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.072526+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:17.072526+0000","last_peered":"2024-09-10T20:21:17.072526+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:17.072526+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3538,"log_dups_size":0,"ondisk_log_size":3538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28955808,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":28285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1573}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3548,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.829402+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:17.829402+0000","last_peered":"2024-09-10T20:21:17.829402+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:17.829402+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457916608,"num_objects":55903,"num_object_clones":0,"num_object_copies":223612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55930,"num_write_kb":447322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":791031808,"data_stored":791031808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55957,"ondisk_log_size":55957,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400188,"kb_used_data":224108,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971652,"statfs":{"total":96636764160,"available":96226971648,"internally_reserved":0,"allocated":229486592,"data_stored":229310611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,3,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398136,"kb_used_data":222568,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93973704,"statfs":{"total":96636764160,"available":96229072896,"internally_reserved":0,"allocated":227909632,"data_stored":227733651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227700,"kb_used_data":101604,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94144140,"statfs":{"total":96636764160,"available":96403599360,"internally_reserved":0,"allocated":104042496,"data_stored":103866515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400124,"kb_used_data":224108,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971716,"statfs":{"total":96636764160,"available":96227037184,"internally_reserved":0,"allocated":229486592,"data_stored":229310611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104763392,"data_stored":104763392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227835904,"data_stored":227835904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:21.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:21.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:21:21.781458+0000 min lat: 0.00230373 max lat: 2.76361 avg lat: 0.0343759 2024-09-10T20:21:21.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:21:21.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 100 20 55960 55940 4.36968 0 - 0.0343759 2024-09-10T20:21:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:21.987 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:21.999 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-10T20:21:21.912736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457916608,"num_objects":55903,"num_object_clones":0,"num_object_copies":223612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55930,"num_write_kb":447322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55957,"ondisk_log_size":55957,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1426148,"kb_used_data":772388,"kb_used_omap":14,"kb_used_meta":653681,"kb_avail":376061212,"statfs":{"total":386547056640,"available":385086681088,"internally_reserved":0,"allocated":790925312,"data_stored":790221388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":669369910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,7,7,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27983872,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":27328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005066"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3523,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.955989+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:17.955989+0000","last_peered":"2024-09-10T20:21:17.955989+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:17.955989+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":3580,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.979163+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:17.979163+0000","last_peered":"2024-09-10T20:21:17.979163+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:17.979163+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3449","reported_seq":3525,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.013892+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:17.013892+0000","last_peered":"2024-09-10T20:21:17.013892+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:17.013892+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28228176,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3663,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.905169+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:17.905169+0000","last_peered":"2024-09-10T20:21:17.905169+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:17.905169+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3581,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.969381+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:17.969381+0000","last_peered":"2024-09-10T20:21:17.969381+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:17.969381+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3572,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:18.027497+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:18.027497+0000","last_peered":"2024-09-10T20:21:18.027497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:18.027497+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3574,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.962718+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:17.962718+0000","last_peered":"2024-09-10T20:21:17.962718+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:17.962718+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3616,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.974792+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:17.974792+0000","last_peered":"2024-09-10T20:21:17.974792+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:17.974792+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":6781,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:19.531780+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:19.531780+0000","last_peered":"2024-09-10T20:21:19.531780+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:19.531780+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28166464,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3444,"num_write_kb":27529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1642,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3383","reported_seq":3460,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:16.998448+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:16.998448+0000","last_peered":"2024-09-10T20:21:16.998448+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:16.998448+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3383,"log_dups_size":0,"ondisk_log_size":3383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694240,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3381,"num_write_kb":27046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3528,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.957343+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:17.957343+0000","last_peered":"2024-09-10T20:21:17.957343+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:17.957343+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.912389+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:17.912389+0000","last_peered":"2024-09-10T20:21:17.912389+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:17.912389+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3404","reported_seq":3478,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.002218+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:17.002218+0000","last_peered":"2024-09-10T20:21:17.002218+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:17.002218+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3404,"log_dups_size":0,"ondisk_log_size":3404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27875920,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3403,"num_write_kb":27223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3579,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.937413+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:17.937413+0000","last_peered":"2024-09-10T20:21:17.937413+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:17.937413+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3538","reported_seq":3616,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.072526+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:17.072526+0000","last_peered":"2024-09-10T20:21:17.072526+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:17.072526+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3538,"log_dups_size":0,"ondisk_log_size":3538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28955808,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":28285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1573}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3548,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.829402+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:17.829402+0000","last_peered":"2024-09-10T20:21:17.829402+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:17.829402+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457916608,"num_objects":55903,"num_object_clones":0,"num_object_copies":223612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55930,"num_write_kb":447322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":791031808,"data_stored":791031808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55957,"ondisk_log_size":55957,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400188,"kb_used_data":224108,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971652,"statfs":{"total":96636764160,"available":96226971648,"internally_reserved":0,"allocated":229486592,"data_stored":229310611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,3,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398136,"kb_used_data":222568,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93973704,"statfs":{"total":96636764160,"available":96229072896,"internally_reserved":0,"allocated":227909632,"data_stored":227733651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227700,"kb_used_data":101604,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94144140,"statfs":{"total":96636764160,"available":96403599360,"internally_reserved":0,"allocated":104042496,"data_stored":103866515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400124,"kb_used_data":224108,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971716,"statfs":{"total":96636764160,"available":96227037184,"internally_reserved":0,"allocated":229486592,"data_stored":229310611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104763392,"data_stored":104763392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227835904,"data_stored":227835904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:22.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:22.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3482", 'reported_seq': 3523, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.955989+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:17.955989+0000', 'last_peered': '2024-09-10T20:21:17.955989+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:17.955989+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3482, 'log_dups_size': 0, 'ondisk_log_size': 3482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28524544, 'num_objects': 3482, 'num_object_clones': 0, 'num_object_copies': 13928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3482, 'num_write_kb': 27856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:22.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:21:22.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3517", 'reported_seq': 3580, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.979163+0000', 'last_change': '2024-09-10T20:20:16.114066+0000', 'last_active': '2024-09-10T20:21:17.979163+0000', 'last_peered': '2024-09-10T20:21:17.979163+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:21:17.979163+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3517, 'log_dups_size': 0, 'ondisk_log_size': 3517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28775584, 'num_objects': 3513, 'num_object_clones': 0, 'num_object_copies': 14052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3515, 'num_write_kb': 28116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:22.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:22.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3449", 'reported_seq': 3525, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.013892+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:17.013892+0000', 'last_peered': '2024-09-10T20:21:17.013892+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:17.013892+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3449, 'log_dups_size': 0, 'ondisk_log_size': 3449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28228176, 'num_objects': 3446, 'num_object_clones': 0, 'num_object_copies': 13784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3448, 'num_write_kb': 27574, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1484}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3601", 'reported_seq': 3663, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.905169+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:21:17.905169+0000', 'last_peered': '2024-09-10T20:21:17.905169+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:17.905169+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3601, 'log_dups_size': 0, 'ondisk_log_size': 3601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29463712, 'num_objects': 3597, 'num_object_clones': 0, 'num_object_copies': 14388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3599, 'num_write_kb': 28781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3530", 'reported_seq': 3581, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.969381+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:17.969381+0000', 'last_peered': '2024-09-10T20:21:17.969381+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:17.969381+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3530, 'log_dups_size': 0, 'ondisk_log_size': 3530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28890272, 'num_objects': 3527, 'num_object_clones': 0, 'num_object_copies': 14108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3528, 'num_write_kb': 28221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1547}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3531", 'reported_seq': 3572, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:18.027497+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:18.027497+0000', 'last_peered': '2024-09-10T20:21:18.027497+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:18.027497+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3531, 'log_dups_size': 0, 'ondisk_log_size': 3531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28925952, 'num_objects': 3531, 'num_object_clones': 0, 'num_object_copies': 14124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3531, 'num_write_kb': 28248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1593}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3510", 'reported_seq': 3574, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.962718+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:17.962718+0000', 'last_peered': '2024-09-10T20:21:17.962718+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:17.962718+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3510, 'log_dups_size': 0, 'ondisk_log_size': 3510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28701856, 'num_objects': 3504, 'num_object_clones': 0, 'num_object_copies': 14016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3508, 'num_write_kb': 28044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1551}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3553", 'reported_seq': 3616, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.974792+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:17.974792+0000', 'last_peered': '2024-09-10T20:21:17.974792+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:17.974792+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3553, 'log_dups_size': 0, 'ondisk_log_size': 3553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29062304, 'num_objects': 3548, 'num_object_clones': 0, 'num_object_copies': 14192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3551, 'num_write_kb': 28396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:22.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3447", 'reported_seq': 6781, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:19.531780+0000', 'last_change': '2024-09-10T20:20:16.102228+0000', 'last_active': '2024-09-10T20:21:19.531780+0000', 'last_peered': '2024-09-10T20:21:19.531780+0000', 'last_clean': '2024-09-10T20:19:06.018547+0000', 'last_became_active': '2024-09-10T20:20:16.083272+0000', 'last_became_peered': '2024-09-10T20:20:16.083272+0000', 'last_unstale': '2024-09-10T20:21:19.531780+0000', 'last_undegraded': '2024-09-10T20:20:15.515842+0000', 'last_fullsized': '2024-09-10T20:20:15.455406+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3447, 'log_dups_size': 0, 'ondisk_log_size': 3447, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28166464, 'num_objects': 3439, 'num_object_clones': 0, 'num_object_copies': 13756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3444, 'num_write_kb': 27529, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1642, 'num_bytes_recovered': 13402112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3155}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3383", 'reported_seq': 3460, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:16.998448+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:16.998448+0000', 'last_peered': '2024-09-10T20:21:16.998448+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:16.998448+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3383, 'log_dups_size': 0, 'ondisk_log_size': 3383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27694240, 'num_objects': 3381, 'num_object_clones': 0, 'num_object_copies': 13524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3381, 'num_write_kb': 27046, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1478}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3474", 'reported_seq': 3528, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.957343+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:17.957343+0000', 'last_peered': '2024-09-10T20:21:17.957343+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:17.957343+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3474, 'log_dups_size': 0, 'ondisk_log_size': 3474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28413680, 'num_objects': 3469, 'num_object_clones': 0, 'num_object_copies': 13876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3471, 'num_write_kb': 27756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3524", 'reported_seq': 3567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.912389+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:17.912389+0000', 'last_peered': '2024-09-10T20:21:17.912389+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:17.912389+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3524, 'log_dups_size': 0, 'ondisk_log_size': 3524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28858960, 'num_objects': 3523, 'num_object_clones': 0, 'num_object_copies': 14092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3523, 'num_write_kb': 28183, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3404", 'reported_seq': 3478, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.002218+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:17.002218+0000', 'last_peered': '2024-09-10T20:21:17.002218+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:17.002218+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3404, 'log_dups_size': 0, 'ondisk_log_size': 3404, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27875920, 'num_objects': 3403, 'num_object_clones': 0, 'num_object_copies': 13612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3403, 'num_write_kb': 27223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1455}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:22.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3529", 'reported_seq': 3579, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.937413+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:17.937413+0000', 'last_peered': '2024-09-10T20:21:17.937413+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:17.937413+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3529, 'log_dups_size': 0, 'ondisk_log_size': 3529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28873888, 'num_objects': 3525, 'num_object_clones': 0, 'num_object_copies': 14100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3527, 'num_write_kb': 28212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:22.003 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:22.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3538", 'reported_seq': 3616, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.072526+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:17.072526+0000', 'last_peered': '2024-09-10T20:21:17.072526+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:17.072526+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3538, 'log_dups_size': 0, 'ondisk_log_size': 3538, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28955808, 'num_objects': 3535, 'num_object_clones': 0, 'num_object_copies': 14140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3536, 'num_write_kb': 28285, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1573}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:22.003 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:22.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3485", 'reported_seq': 3548, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:17.829402+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:17.829402+0000', 'last_peered': '2024-09-10T20:21:17.829402+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:17.829402+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3485, 'log_dups_size': 0, 'ondisk_log_size': 3485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28505248, 'num_objects': 3480, 'num_object_clones': 0, 'num_object_copies': 13920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3483, 'num_write_kb': 27852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:22.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:22.475 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:22.475 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29845/223612 objects degraded (13.347%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":171,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55903,"data_bytes":457916608,"bytes_used":1460375552,"bytes_avail":385086681088,"bytes_total":386547056640,"degraded_objects":29845,"degraded_total":223612,"degraded_ratio":0.13346779242616674,"recovering_objects_per_sec":59,"recovering_bytes_per_sec":486358,"recovering_keys_per_sec":0,"num_objects_recovered":594,"num_bytes_recovered":4866048,"num_keys_recovered":0,"write_bytes_sec":2796970,"read_op_per_sec":0,"write_op_per_sec":341},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-10T20:21:13.910600+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:21:22.494 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:21:22.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:22.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 101 20 55960 55940 4.32642 0 - 0.0343759 2024-09-10T20:21:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:22.842 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:22.854 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-10T20:21:22.141363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458612928,"num_objects":55988,"num_object_clones":0,"num_object_copies":223952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56015,"num_write_kb":448002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1427700,"kb_used_data":773936,"kb_used_omap":14,"kb_used_meta":653681,"kb_avail":376059660,"statfs":{"total":386547056640,"available":385085091840,"internally_reserved":0,"allocated":792510464,"data_stored":791801971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":669369910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,5,6,3,5,2,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.230834"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3523,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.955989+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:17.955989+0000","last_peered":"2024-09-10T20:21:17.955989+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:17.955989+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":3580,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.979163+0000","last_change":"2024-09-10T20:20:16.114066+0000","last_active":"2024-09-10T20:21:17.979163+0000","last_peered":"2024-09-10T20:21:17.979163+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:17.979163+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3467","reported_seq":3542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.142573+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:21.142573+0000","last_peered":"2024-09-10T20:21:21.142573+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:21.142573+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28375632,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3663,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.905169+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:17.905169+0000","last_peered":"2024-09-10T20:21:17.905169+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:17.905169+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3581,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.969381+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:17.969381+0000","last_peered":"2024-09-10T20:21:17.969381+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:17.969381+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3572,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:18.027497+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:18.027497+0000","last_peered":"2024-09-10T20:21:18.027497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:18.027497+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3574,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.962718+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:17.962718+0000","last_peered":"2024-09-10T20:21:17.962718+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:17.962718+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3616,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.974792+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:17.974792+0000","last_peered":"2024-09-10T20:21:17.974792+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:17.974792+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":6781,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:19.531780+0000","last_change":"2024-09-10T20:20:16.102228+0000","last_active":"2024-09-10T20:21:19.531780+0000","last_peered":"2024-09-10T20:21:19.531780+0000","last_clean":"2024-09-10T20:19:06.018547+0000","last_became_active":"2024-09-10T20:20:16.083272+0000","last_became_peered":"2024-09-10T20:20:16.083272+0000","last_unstale":"2024-09-10T20:21:19.531780+0000","last_undegraded":"2024-09-10T20:20:15.515842+0000","last_fullsized":"2024-09-10T20:20:15.455406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28166464,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3444,"num_write_kb":27529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1642,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3406","reported_seq":3480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.143615+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:21.143615+0000","last_peered":"2024-09-10T20:21:21.143615+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:21.143615+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27882656,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3528,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.957343+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:17.957343+0000","last_peered":"2024-09-10T20:21:17.957343+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:17.957343+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.912389+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:17.912389+0000","last_peered":"2024-09-10T20:21:17.912389+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:17.912389+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3425","reported_seq":3496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.146888+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:21.146888+0000","last_peered":"2024-09-10T20:21:21.146888+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:21.146888+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3579,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.937413+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:17.937413+0000","last_peered":"2024-09-10T20:21:17.937413+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:17.937413+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3561","reported_seq":3636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.147998+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:21.147998+0000","last_peered":"2024-09-10T20:21:21.147998+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:21.147998+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144224,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3548,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:17.829402+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:17.829402+0000","last_peered":"2024-09-10T20:21:17.829402+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:17.829402+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":458612928,"num_objects":55988,"num_object_clones":0,"num_object_copies":223952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56015,"num_write_kb":448002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":13402112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":792412160,"data_stored":792412160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400188,"kb_used_data":224108,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971652,"statfs":{"total":96636764160,"available":96226971648,"internally_reserved":0,"allocated":229486592,"data_stored":229310611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,3,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":224116,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":229494784,"data_stored":229314234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227700,"kb_used_data":101604,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94144140,"statfs":{"total":96636764160,"available":96403599360,"internally_reserved":0,"allocated":104042496,"data_stored":103866515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400124,"kb_used_data":224108,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971716,"statfs":{"total":96636764160,"available":96227037184,"internally_reserved":0,"allocated":229486592,"data_stored":229310611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104763392,"data_stored":104763392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:23.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 102 20 55960 55940 4.284 0 - 0.0343759 2024-09-10T20:21:24.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 103 20 55960 55940 4.24241 0 - 0.0343759 2024-09-10T20:21:25.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 104 20 55960 55940 4.20162 0 - 0.0343759 2024-09-10T20:21:25.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:26.190 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:26.191 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:26.203 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-10T20:21:25.913593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1437412,"kb_used_data":779468,"kb_used_omap":14,"kb_used_meta":657841,"kb_avail":376049948,"statfs":{"total":386547056640,"available":385075146752,"internally_reserved":0,"allocated":798175232,"data_stored":797439262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":673629750},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9072912,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2334,"num_write_kb":-18649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-542,"num_bytes_recovered":-4390912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001898"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3527,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.153417+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:22.153417+0000","last_peered":"2024-09-10T20:21:22.153417+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:22.153417+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":4763,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:24.260840+0000","last_change":"2024-09-10T20:21:20.297752+0000","last_active":"2024-09-10T20:21:24.260840+0000","last_peered":"2024-09-10T20:21:24.260840+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:24.260840+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3467","reported_seq":3542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.142573+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:21.142573+0000","last_peered":"2024-09-10T20:21:21.142573+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:21.142573+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28375632,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3668,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.157338+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:22.157338+0000","last_peered":"2024-09-10T20:21:22.157338+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:22.157338+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3585,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148469+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:22.148469+0000","last_peered":"2024-09-10T20:21:22.148469+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:22.148469+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3576,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149186+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:22.149186+0000","last_peered":"2024-09-10T20:21:22.149186+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:22.149186+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3579,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:22.149168+0000","last_peered":"2024-09-10T20:21:22.149168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:22.149168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3620,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144554+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:22.144554+0000","last_peered":"2024-09-10T20:21:22.144554+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:22.144554+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":7404,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-10T20:21:22.722881+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:22.722881+0000","last_peered":"2024-09-10T20:21:22.722881+0000","last_clean":"2024-09-10T20:21:22.722881+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:22.722881+0000","last_undegraded":"2024-09-10T20:21:22.722881+0000","last_fullsized":"2024-09-10T20:21:22.722881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28146256,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3406","reported_seq":3480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.143615+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:21.143615+0000","last_peered":"2024-09-10T20:21:21.143615+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:21.143615+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27882656,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3532,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148389+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:22.148389+0000","last_peered":"2024-09-10T20:21:22.148389+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:22.148389+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3571,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.154320+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:22.154320+0000","last_peered":"2024-09-10T20:21:22.154320+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:22.154320+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3425","reported_seq":3496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.146888+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:21.146888+0000","last_peered":"2024-09-10T20:21:21.146888+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:21.146888+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3583,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148982+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:22.148982+0000","last_peered":"2024-09-10T20:21:22.148982+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:22.148982+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3561","reported_seq":3636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.147998+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:21.147998+0000","last_peered":"2024-09-10T20:21:21.147998+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:21.147998+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144224,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3552,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.150318+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:22.150318+0000","last_peered":"2024-09-10T20:21:22.150318+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:22.150318+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":797298688,"data_stored":797298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400208,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971632,"statfs":{"total":96636764160,"available":96226951168,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":224116,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":229494784,"data_stored":229314234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237368,"kb_used_data":107104,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94134472,"statfs":{"total":96636764160,"available":96393699328,"internally_reserved":0,"allocated":109674496,"data_stored":109489356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400148,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971692,"statfs":{"total":96636764160,"available":96227012608,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:26.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:26.526 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:26.527 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:26.539 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-10T20:21:25.913593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1437412,"kb_used_data":779468,"kb_used_omap":14,"kb_used_meta":657841,"kb_avail":376049948,"statfs":{"total":386547056640,"available":385075146752,"internally_reserved":0,"allocated":798175232,"data_stored":797439262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":673629750},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9072912,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2334,"num_write_kb":-18649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-542,"num_bytes_recovered":-4390912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001898"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3527,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.153417+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:22.153417+0000","last_peered":"2024-09-10T20:21:22.153417+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:22.153417+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":4763,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:24.260840+0000","last_change":"2024-09-10T20:21:20.297752+0000","last_active":"2024-09-10T20:21:24.260840+0000","last_peered":"2024-09-10T20:21:24.260840+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:24.260840+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3467","reported_seq":3542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.142573+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:21.142573+0000","last_peered":"2024-09-10T20:21:21.142573+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:21.142573+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28375632,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3668,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.157338+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:22.157338+0000","last_peered":"2024-09-10T20:21:22.157338+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:22.157338+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3585,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148469+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:22.148469+0000","last_peered":"2024-09-10T20:21:22.148469+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:22.148469+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3576,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149186+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:22.149186+0000","last_peered":"2024-09-10T20:21:22.149186+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:22.149186+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3579,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:22.149168+0000","last_peered":"2024-09-10T20:21:22.149168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:22.149168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3620,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144554+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:22.144554+0000","last_peered":"2024-09-10T20:21:22.144554+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:22.144554+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":7404,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-10T20:21:22.722881+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:22.722881+0000","last_peered":"2024-09-10T20:21:22.722881+0000","last_clean":"2024-09-10T20:21:22.722881+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:22.722881+0000","last_undegraded":"2024-09-10T20:21:22.722881+0000","last_fullsized":"2024-09-10T20:21:22.722881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28146256,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3406","reported_seq":3480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.143615+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:21.143615+0000","last_peered":"2024-09-10T20:21:21.143615+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:21.143615+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27882656,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3532,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148389+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:22.148389+0000","last_peered":"2024-09-10T20:21:22.148389+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:22.148389+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3571,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.154320+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:22.154320+0000","last_peered":"2024-09-10T20:21:22.154320+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:22.154320+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3425","reported_seq":3496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.146888+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:21.146888+0000","last_peered":"2024-09-10T20:21:21.146888+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:21.146888+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3583,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148982+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:22.148982+0000","last_peered":"2024-09-10T20:21:22.148982+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:22.148982+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3561","reported_seq":3636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.147998+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:21.147998+0000","last_peered":"2024-09-10T20:21:21.147998+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:21.147998+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144224,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3552,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.150318+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:22.150318+0000","last_peered":"2024-09-10T20:21:22.150318+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:22.150318+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":797298688,"data_stored":797298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400208,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971632,"statfs":{"total":96636764160,"available":96226951168,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":224116,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":229494784,"data_stored":229314234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237368,"kb_used_data":107104,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94134472,"statfs":{"total":96636764160,"available":96393699328,"internally_reserved":0,"allocated":109674496,"data_stored":109489356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400148,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971692,"statfs":{"total":96636764160,"available":96227012608,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3482", 'reported_seq': 3527, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.153417+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:22.153417+0000', 'last_peered': '2024-09-10T20:21:22.153417+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:22.153417+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3482, 'log_dups_size': 0, 'ondisk_log_size': 3482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28524544, 'num_objects': 3482, 'num_object_clones': 0, 'num_object_copies': 13928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3482, 'num_write_kb': 27856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3517", 'reported_seq': 4763, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:24.260840+0000', 'last_change': '2024-09-10T20:21:20.297752+0000', 'last_active': '2024-09-10T20:21:24.260840+0000', 'last_peered': '2024-09-10T20:21:24.260840+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:21:24.260840+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3517, 'log_dups_size': 0, 'ondisk_log_size': 3517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28775584, 'num_objects': 3513, 'num_object_clones': 0, 'num_object_copies': 14052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1402, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3515, 'num_write_kb': 28116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 591, 'num_bytes_recovered': 4825088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3467", 'reported_seq': 3542, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:21.142573+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:21.142573+0000', 'last_peered': '2024-09-10T20:21:21.142573+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:21.142573+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3467, 'log_dups_size': 0, 'ondisk_log_size': 3467, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28375632, 'num_objects': 3464, 'num_object_clones': 0, 'num_object_copies': 13856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3466, 'num_write_kb': 27718, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1502}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:26.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3601", 'reported_seq': 3668, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.157338+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:21:22.157338+0000', 'last_peered': '2024-09-10T20:21:22.157338+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:22.157338+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3601, 'log_dups_size': 0, 'ondisk_log_size': 3601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29463712, 'num_objects': 3597, 'num_object_clones': 0, 'num_object_copies': 14388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3599, 'num_write_kb': 28781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3530", 'reported_seq': 3585, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.148469+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:22.148469+0000', 'last_peered': '2024-09-10T20:21:22.148469+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:22.148469+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3530, 'log_dups_size': 0, 'ondisk_log_size': 3530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28890272, 'num_objects': 3527, 'num_object_clones': 0, 'num_object_copies': 14108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3528, 'num_write_kb': 28221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1547}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3531", 'reported_seq': 3576, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.149186+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:22.149186+0000', 'last_peered': '2024-09-10T20:21:22.149186+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:22.149186+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3531, 'log_dups_size': 0, 'ondisk_log_size': 3531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28925952, 'num_objects': 3531, 'num_object_clones': 0, 'num_object_copies': 14124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3531, 'num_write_kb': 28248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1593}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3510", 'reported_seq': 3579, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.149168+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:22.149168+0000', 'last_peered': '2024-09-10T20:21:22.149168+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:22.149168+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3510, 'log_dups_size': 0, 'ondisk_log_size': 3510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28701856, 'num_objects': 3504, 'num_object_clones': 0, 'num_object_copies': 14016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3508, 'num_write_kb': 28044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1551}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3553", 'reported_seq': 3620, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.144554+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:22.144554+0000', 'last_peered': '2024-09-10T20:21:22.144554+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:22.144554+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3553, 'log_dups_size': 0, 'ondisk_log_size': 3553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29062304, 'num_objects': 3548, 'num_object_clones': 0, 'num_object_copies': 14192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3551, 'num_write_kb': 28396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3406", 'reported_seq': 3480, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:21.143615+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:21.143615+0000', 'last_peered': '2024-09-10T20:21:21.143615+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:21.143615+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3406, 'log_dups_size': 0, 'ondisk_log_size': 3406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27882656, 'num_objects': 3404, 'num_object_clones': 0, 'num_object_copies': 13616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3404, 'num_write_kb': 27230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1501}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:26.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3474", 'reported_seq': 3532, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.148389+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:22.148389+0000', 'last_peered': '2024-09-10T20:21:22.148389+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:22.148389+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3474, 'log_dups_size': 0, 'ondisk_log_size': 3474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28413680, 'num_objects': 3469, 'num_object_clones': 0, 'num_object_copies': 13876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3471, 'num_write_kb': 27756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3524", 'reported_seq': 3571, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.154320+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:22.154320+0000', 'last_peered': '2024-09-10T20:21:22.154320+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:22.154320+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3524, 'log_dups_size': 0, 'ondisk_log_size': 3524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28858960, 'num_objects': 3523, 'num_object_clones': 0, 'num_object_copies': 14092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3523, 'num_write_kb': 28183, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3425", 'reported_seq': 3496, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:21.146888+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:21.146888+0000', 'last_peered': '2024-09-10T20:21:21.146888+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:21.146888+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3425, 'log_dups_size': 0, 'ondisk_log_size': 3425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28047952, 'num_objects': 3424, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3424, 'num_write_kb': 27391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3529", 'reported_seq': 3583, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.148982+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:22.148982+0000', 'last_peered': '2024-09-10T20:21:22.148982+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:22.148982+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3529, 'log_dups_size': 0, 'ondisk_log_size': 3529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28873888, 'num_objects': 3525, 'num_object_clones': 0, 'num_object_copies': 14100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3527, 'num_write_kb': 28212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3561", 'reported_seq': 3636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:21.147998+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:21.147998+0000', 'last_peered': '2024-09-10T20:21:21.147998+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:21.147998+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3561, 'log_dups_size': 0, 'ondisk_log_size': 3561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29144224, 'num_objects': 3558, 'num_object_clones': 0, 'num_object_copies': 14232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3559, 'num_write_kb': 28469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1596}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:26.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3485", 'reported_seq': 3552, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.150318+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:22.150318+0000', 'last_peered': '2024-09-10T20:21:22.150318+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:22.150318+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3485, 'log_dups_size': 0, 'ondisk_log_size': 3485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28505248, 'num_objects': 3480, 'num_object_clones': 0, 'num_object_copies': 13920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3483, 'num_write_kb': 27852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:26.543 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:26.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 105 20 55960 55940 4.16161 0 - 0.0343759 2024-09-10T20:21:26.962 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:26.962 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28970/223940 objects degraded (12.937%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":175,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55985,"data_bytes":458592720,"bytes_used":1471909888,"bytes_avail":385075146752,"bytes_total":386547056640,"degraded_objects":28970,"degraded_total":223940,"degraded_ratio":0.12936500848441546},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-10T20:21:13.910600+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:21:26.975 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:21:26.975 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:27.289 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:27.301 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-10T20:21:25.913593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1437412,"kb_used_data":779468,"kb_used_omap":14,"kb_used_meta":657841,"kb_avail":376049948,"statfs":{"total":386547056640,"available":385075146752,"internally_reserved":0,"allocated":798175232,"data_stored":797439262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":673629750},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9072912,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2334,"num_write_kb":-18649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-542,"num_bytes_recovered":-4390912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001898"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3527,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.153417+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:22.153417+0000","last_peered":"2024-09-10T20:21:22.153417+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:22.153417+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":4763,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:24.260840+0000","last_change":"2024-09-10T20:21:20.297752+0000","last_active":"2024-09-10T20:21:24.260840+0000","last_peered":"2024-09-10T20:21:24.260840+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:24.260840+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3467","reported_seq":3542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.142573+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:21.142573+0000","last_peered":"2024-09-10T20:21:21.142573+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:21.142573+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28375632,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3668,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.157338+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:22.157338+0000","last_peered":"2024-09-10T20:21:22.157338+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:22.157338+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3585,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148469+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:22.148469+0000","last_peered":"2024-09-10T20:21:22.148469+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:22.148469+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3576,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149186+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:22.149186+0000","last_peered":"2024-09-10T20:21:22.149186+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:22.149186+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3579,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:22.149168+0000","last_peered":"2024-09-10T20:21:22.149168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:22.149168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3620,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144554+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:22.144554+0000","last_peered":"2024-09-10T20:21:22.144554+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:22.144554+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":7404,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-10T20:21:22.722881+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:22.722881+0000","last_peered":"2024-09-10T20:21:22.722881+0000","last_clean":"2024-09-10T20:21:22.722881+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:22.722881+0000","last_undegraded":"2024-09-10T20:21:22.722881+0000","last_fullsized":"2024-09-10T20:21:22.722881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28146256,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3406","reported_seq":3480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.143615+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:21.143615+0000","last_peered":"2024-09-10T20:21:21.143615+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:21.143615+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27882656,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3532,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148389+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:22.148389+0000","last_peered":"2024-09-10T20:21:22.148389+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:22.148389+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3571,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.154320+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:22.154320+0000","last_peered":"2024-09-10T20:21:22.154320+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:22.154320+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3425","reported_seq":3496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.146888+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:21.146888+0000","last_peered":"2024-09-10T20:21:21.146888+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:21.146888+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3583,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148982+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:22.148982+0000","last_peered":"2024-09-10T20:21:22.148982+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:22.148982+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3561","reported_seq":3636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:21.147998+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:21.147998+0000","last_peered":"2024-09-10T20:21:21.147998+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:21.147998+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144224,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3552,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.150318+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:22.150318+0000","last_peered":"2024-09-10T20:21:22.150318+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:22.150318+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":4825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":797298688,"data_stored":797298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400208,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971632,"statfs":{"total":96636764160,"available":96226951168,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":224116,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":229494784,"data_stored":229314234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,20],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237368,"kb_used_data":107104,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94134472,"statfs":{"total":96636764160,"available":96393699328,"internally_reserved":0,"allocated":109674496,"data_stored":109489356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400148,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971692,"statfs":{"total":96636764160,"available":96227012608,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:27.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 106 20 55960 55940 4.12235 0 - 0.0343759 2024-09-10T20:21:28.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 107 20 55960 55940 4.08383 0 - 0.0343759 2024-09-10T20:21:29.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 108 20 55960 55940 4.04602 0 - 0.0343759 2024-09-10T20:21:30.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:30.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 109 20 56213 56193 4.02703 0.164714 0.0281586 0.0387054 2024-09-10T20:21:30.879 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:30.880 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:30.904 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-10T20:21:30.025007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1428616,"kb_used_data":783156,"kb_used_omap":14,"kb_used_meta":645361,"kb_avail":376058744,"statfs":{"total":386547056640,"available":385084153856,"internally_reserved":0,"allocated":801951744,"data_stored":801211184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":660850230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":676112,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3359,"num_write_kb":-26849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.112271"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3527,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.153417+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:22.153417+0000","last_peered":"2024-09-10T20:21:22.153417+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:22.153417+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":7197,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:29.261539+0000","last_change":"2024-09-10T20:21:20.297752+0000","last_active":"2024-09-10T20:21:29.261539+0000","last_peered":"2024-09-10T20:21:29.261539+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:29.261539+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1808,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3467","reported_seq":3544,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144014+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:22.144014+0000","last_peered":"2024-09-10T20:21:22.144014+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:22.144014+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28375632,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3668,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.157338+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:22.157338+0000","last_peered":"2024-09-10T20:21:22.157338+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:22.157338+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3585,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148469+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:22.148469+0000","last_peered":"2024-09-10T20:21:22.148469+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:22.148469+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3576,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149186+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:22.149186+0000","last_peered":"2024-09-10T20:21:22.149186+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:22.149186+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3579,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:22.149168+0000","last_peered":"2024-09-10T20:21:22.149168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:22.149168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3621,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149150+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:22.149150+0000","last_peered":"2024-09-10T20:21:22.149150+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:22.149150+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":7404,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-10T20:21:22.722881+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:22.722881+0000","last_peered":"2024-09-10T20:21:22.722881+0000","last_clean":"2024-09-10T20:21:22.722881+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:22.722881+0000","last_undegraded":"2024-09-10T20:21:22.722881+0000","last_fullsized":"2024-09-10T20:21:22.722881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28146256,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3406","reported_seq":3482,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144057+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:22.144057+0000","last_peered":"2024-09-10T20:21:22.144057+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:22.144057+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27882656,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3532,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148389+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:22.148389+0000","last_peered":"2024-09-10T20:21:22.148389+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:22.148389+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3571,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.154320+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:22.154320+0000","last_peered":"2024-09-10T20:21:22.154320+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:22.154320+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3425","reported_seq":3498,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.143918+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:22.143918+0000","last_peered":"2024-09-10T20:21:22.143918+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:22.143918+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3583,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148982+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:22.148982+0000","last_peered":"2024-09-10T20:21:22.148982+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:22.148982+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3561","reported_seq":3638,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.143643+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:22.143643+0000","last_peered":"2024-09-10T20:21:22.143643+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:22.143643+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144224,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3552,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.150318+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:22.150318+0000","last_peered":"2024-09-10T20:21:22.150318+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:22.150318+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":801415168,"data_stored":801415168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400208,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971632,"statfs":{"total":96636764160,"available":96226951168,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399700,"kb_used_data":224124,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93972140,"statfs":{"total":96636764160,"available":96227471360,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228560,"kb_used_data":110784,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94143280,"statfs":{"total":96636764160,"available":96402718720,"internally_reserved":0,"allocated":113442816,"data_stored":113257676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120581842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400148,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971692,"statfs":{"total":96636764160,"available":96227012608,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113766400,"data_stored":113766400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:30.907 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:31.474 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:31.474 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:31.503 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-10T20:21:31.032533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1428616,"kb_used_data":783156,"kb_used_omap":14,"kb_used_meta":645361,"kb_avail":376058744,"statfs":{"total":386547056640,"available":385084153856,"internally_reserved":0,"allocated":801951744,"data_stored":801211184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":660850230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3444,"num_write_kb":-27529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.891170"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3527,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.153417+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:22.153417+0000","last_peered":"2024-09-10T20:21:22.153417+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:22.153417+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":7197,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:29.261539+0000","last_change":"2024-09-10T20:21:20.297752+0000","last_active":"2024-09-10T20:21:29.261539+0000","last_peered":"2024-09-10T20:21:29.261539+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:29.261539+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1808,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3467","reported_seq":3544,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144014+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:22.144014+0000","last_peered":"2024-09-10T20:21:22.144014+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:22.144014+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28375632,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3668,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.157338+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:22.157338+0000","last_peered":"2024-09-10T20:21:22.157338+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:22.157338+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3585,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148469+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:22.148469+0000","last_peered":"2024-09-10T20:21:22.148469+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:22.148469+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3576,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149186+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:22.149186+0000","last_peered":"2024-09-10T20:21:22.149186+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:22.149186+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3579,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:22.149168+0000","last_peered":"2024-09-10T20:21:22.149168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:22.149168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3621,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149150+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:22.149150+0000","last_peered":"2024-09-10T20:21:22.149150+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:22.149150+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":7404,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-10T20:21:22.722881+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:22.722881+0000","last_peered":"2024-09-10T20:21:22.722881+0000","last_clean":"2024-09-10T20:21:22.722881+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:22.722881+0000","last_undegraded":"2024-09-10T20:21:22.722881+0000","last_fullsized":"2024-09-10T20:21:22.722881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28146256,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3406","reported_seq":3482,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144057+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:22.144057+0000","last_peered":"2024-09-10T20:21:22.144057+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:22.144057+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27882656,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3532,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148389+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:22.148389+0000","last_peered":"2024-09-10T20:21:22.148389+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:22.148389+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3571,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.154320+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:22.154320+0000","last_peered":"2024-09-10T20:21:22.154320+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:22.154320+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3425","reported_seq":3498,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.143918+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:22.143918+0000","last_peered":"2024-09-10T20:21:22.143918+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:22.143918+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3583,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148982+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:22.148982+0000","last_peered":"2024-09-10T20:21:22.148982+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:22.148982+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3561","reported_seq":3638,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.143643+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:22.143643+0000","last_peered":"2024-09-10T20:21:22.143643+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:22.143643+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144224,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3552,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.150318+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:22.150318+0000","last_peered":"2024-09-10T20:21:22.150318+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:22.150318+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":801415168,"data_stored":801415168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400208,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971632,"statfs":{"total":96636764160,"available":96226951168,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399700,"kb_used_data":224124,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93972140,"statfs":{"total":96636764160,"available":96227471360,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228560,"kb_used_data":110784,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94143280,"statfs":{"total":96636764160,"available":96402718720,"internally_reserved":0,"allocated":113442816,"data_stored":113257676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120581842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400148,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971692,"statfs":{"total":96636764160,"available":96227012608,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113766400,"data_stored":113766400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3482", 'reported_seq': 3527, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.153417+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:22.153417+0000', 'last_peered': '2024-09-10T20:21:22.153417+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:22.153417+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3482, 'log_dups_size': 0, 'ondisk_log_size': 3482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28524544, 'num_objects': 3482, 'num_object_clones': 0, 'num_object_copies': 13928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3482, 'num_write_kb': 27856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3517", 'reported_seq': 7197, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:29.261539+0000', 'last_change': '2024-09-10T20:21:20.297752+0000', 'last_active': '2024-09-10T20:21:29.261539+0000', 'last_peered': '2024-09-10T20:21:29.261539+0000', 'last_clean': '2024-09-10T20:19:05.871495+0000', 'last_became_active': '2024-09-10T20:20:16.109539+0000', 'last_became_peered': '2024-09-10T20:20:16.109539+0000', 'last_unstale': '2024-09-10T20:21:29.261539+0000', 'last_undegraded': '2024-09-10T20:20:15.497703+0000', 'last_fullsized': '2024-09-10T20:20:15.454500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3517, 'log_dups_size': 0, 'ondisk_log_size': 3517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28775584, 'num_objects': 3513, 'num_object_clones': 0, 'num_object_copies': 14052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 187, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3515, 'num_write_kb': 28116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1808, 'num_bytes_recovered': 14778368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3467", 'reported_seq': 3544, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.144014+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:22.144014+0000', 'last_peered': '2024-09-10T20:21:22.144014+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:22.144014+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3467, 'log_dups_size': 0, 'ondisk_log_size': 3467, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28375632, 'num_objects': 3464, 'num_object_clones': 0, 'num_object_copies': 13856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3466, 'num_write_kb': 27718, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1502}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:31.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3601", 'reported_seq': 3668, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.157338+0000', 'last_change': '2024-09-10T20:20:16.144409+0000', 'last_active': '2024-09-10T20:21:22.157338+0000', 'last_peered': '2024-09-10T20:21:22.157338+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:22.157338+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3601, 'log_dups_size': 0, 'ondisk_log_size': 3601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29463712, 'num_objects': 3597, 'num_object_clones': 0, 'num_object_copies': 14388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3599, 'num_write_kb': 28781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3530", 'reported_seq': 3585, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.148469+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:22.148469+0000', 'last_peered': '2024-09-10T20:21:22.148469+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:22.148469+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3530, 'log_dups_size': 0, 'ondisk_log_size': 3530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28890272, 'num_objects': 3527, 'num_object_clones': 0, 'num_object_copies': 14108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3528, 'num_write_kb': 28221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1547}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3531", 'reported_seq': 3576, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.149186+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:22.149186+0000', 'last_peered': '2024-09-10T20:21:22.149186+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:22.149186+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3531, 'log_dups_size': 0, 'ondisk_log_size': 3531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28925952, 'num_objects': 3531, 'num_object_clones': 0, 'num_object_copies': 14124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3531, 'num_write_kb': 28248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1593}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3510", 'reported_seq': 3579, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.149168+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:22.149168+0000', 'last_peered': '2024-09-10T20:21:22.149168+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:22.149168+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3510, 'log_dups_size': 0, 'ondisk_log_size': 3510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28701856, 'num_objects': 3504, 'num_object_clones': 0, 'num_object_copies': 14016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3508, 'num_write_kb': 28044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1551}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3553", 'reported_seq': 3621, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.149150+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:22.149150+0000', 'last_peered': '2024-09-10T20:21:22.149150+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:22.149150+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3553, 'log_dups_size': 0, 'ondisk_log_size': 3553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29062304, 'num_objects': 3548, 'num_object_clones': 0, 'num_object_copies': 14192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3551, 'num_write_kb': 28396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3406", 'reported_seq': 3482, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.144057+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:22.144057+0000', 'last_peered': '2024-09-10T20:21:22.144057+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:22.144057+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3406, 'log_dups_size': 0, 'ondisk_log_size': 3406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27882656, 'num_objects': 3404, 'num_object_clones': 0, 'num_object_copies': 13616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3404, 'num_write_kb': 27230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1501}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:31.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:31.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3474", 'reported_seq': 3532, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.148389+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:22.148389+0000', 'last_peered': '2024-09-10T20:21:22.148389+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:22.148389+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3474, 'log_dups_size': 0, 'ondisk_log_size': 3474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28413680, 'num_objects': 3469, 'num_object_clones': 0, 'num_object_copies': 13876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3471, 'num_write_kb': 27756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:31.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:31.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3524", 'reported_seq': 3571, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.154320+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:22.154320+0000', 'last_peered': '2024-09-10T20:21:22.154320+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:22.154320+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3524, 'log_dups_size': 0, 'ondisk_log_size': 3524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28858960, 'num_objects': 3523, 'num_object_clones': 0, 'num_object_copies': 14092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3523, 'num_write_kb': 28183, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:31.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:31.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3425", 'reported_seq': 3498, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.143918+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:22.143918+0000', 'last_peered': '2024-09-10T20:21:22.143918+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:22.143918+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3425, 'log_dups_size': 0, 'ondisk_log_size': 3425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28047952, 'num_objects': 3424, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3424, 'num_write_kb': 27391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:31.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:31.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3529", 'reported_seq': 3583, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.148982+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:22.148982+0000', 'last_peered': '2024-09-10T20:21:22.148982+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:22.148982+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3529, 'log_dups_size': 0, 'ondisk_log_size': 3529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28873888, 'num_objects': 3525, 'num_object_clones': 0, 'num_object_copies': 14100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3527, 'num_write_kb': 28212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:31.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:31.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3561", 'reported_seq': 3638, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.143643+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:22.143643+0000', 'last_peered': '2024-09-10T20:21:22.143643+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:22.143643+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3561, 'log_dups_size': 0, 'ondisk_log_size': 3561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29144224, 'num_objects': 3558, 'num_object_clones': 0, 'num_object_copies': 14232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3559, 'num_write_kb': 28469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1596}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:31.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:31.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3485", 'reported_seq': 3552, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:22.150318+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:22.150318+0000', 'last_peered': '2024-09-10T20:21:22.150318+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:22.150318+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3485, 'log_dups_size': 0, 'ondisk_log_size': 3485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28505248, 'num_objects': 3480, 'num_object_clones': 0, 'num_object_copies': 13920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3483, 'num_write_kb': 27852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:31.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:31.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 110 20 56524 56504 4.01251 2.42969 0.0568205 0.0389275 2024-09-10T20:21:32.219 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:32.219 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27755/223940 objects degraded (12.394%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":180,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55985,"data_bytes":458592720,"bytes_used":1462902784,"bytes_avail":385084153856,"bytes_total":386547056640,"degraded_objects":27755,"degraded_total":223940,"degraded_ratio":0.12393944806644637,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":171994,"recovering_keys_per_sec":0,"num_objects_recovered":164,"num_bytes_recovered":1376256,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:21:32.254 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:21:32.254 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:32.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 111 20 56797 56777 3.99557 2.13281 0.052026 0.0390912 2024-09-10T20:21:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:32.845 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:32.870 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-10T20:21:31.914863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1428616,"kb_used_data":783156,"kb_used_omap":14,"kb_used_meta":645361,"kb_avail":376058744,"statfs":{"total":386547056640,"available":385084153856,"internally_reserved":0,"allocated":801951744,"data_stored":801211184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":660850230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3444,"num_write_kb":-27529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001726"},"pg_stats":[{"pgid":"1.f","version":"32'3482","reported_seq":3527,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.153417+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:22.153417+0000","last_peered":"2024-09-10T20:21:22.153417+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:22.153417+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3517","reported_seq":7197,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:29.261539+0000","last_change":"2024-09-10T20:21:20.297752+0000","last_active":"2024-09-10T20:21:29.261539+0000","last_peered":"2024-09-10T20:21:29.261539+0000","last_clean":"2024-09-10T20:19:05.871495+0000","last_became_active":"2024-09-10T20:20:16.109539+0000","last_became_peered":"2024-09-10T20:20:16.109539+0000","last_unstale":"2024-09-10T20:21:29.261539+0000","last_undegraded":"2024-09-10T20:20:15.497703+0000","last_fullsized":"2024-09-10T20:20:15.454500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28775584,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1808,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3467","reported_seq":3544,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144014+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:22.144014+0000","last_peered":"2024-09-10T20:21:22.144014+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:22.144014+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28375632,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3601","reported_seq":3668,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.157338+0000","last_change":"2024-09-10T20:20:16.144409+0000","last_active":"2024-09-10T20:21:22.157338+0000","last_peered":"2024-09-10T20:21:22.157338+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:22.157338+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3530","reported_seq":3585,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148469+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:22.148469+0000","last_peered":"2024-09-10T20:21:22.148469+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:22.148469+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28890272,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3531","reported_seq":3576,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149186+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:22.149186+0000","last_peered":"2024-09-10T20:21:22.149186+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:22.149186+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3510","reported_seq":3579,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:22.149168+0000","last_peered":"2024-09-10T20:21:22.149168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:22.149168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28701856,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3553","reported_seq":3621,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.149150+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:22.149150+0000","last_peered":"2024-09-10T20:21:22.149150+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:22.149150+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3447","reported_seq":7404,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-10T20:21:22.722881+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:22.722881+0000","last_peered":"2024-09-10T20:21:22.722881+0000","last_clean":"2024-09-10T20:21:22.722881+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:22.722881+0000","last_undegraded":"2024-09-10T20:21:22.722881+0000","last_fullsized":"2024-09-10T20:21:22.722881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28146256,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3406","reported_seq":3482,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.144057+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:22.144057+0000","last_peered":"2024-09-10T20:21:22.144057+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:22.144057+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27882656,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3474","reported_seq":3532,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148389+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:22.148389+0000","last_peered":"2024-09-10T20:21:22.148389+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:22.148389+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28413680,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3524","reported_seq":3571,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.154320+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:22.154320+0000","last_peered":"2024-09-10T20:21:22.154320+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:22.154320+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28858960,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3425","reported_seq":3498,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.143918+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:22.143918+0000","last_peered":"2024-09-10T20:21:22.143918+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:22.143918+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3529","reported_seq":3583,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.148982+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:22.148982+0000","last_peered":"2024-09-10T20:21:22.148982+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:22.148982+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873888,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3561","reported_seq":3638,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.143643+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:22.143643+0000","last_peered":"2024-09-10T20:21:22.143643+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:22.143643+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144224,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3485","reported_seq":3552,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:22.150318+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:22.150318+0000","last_peered":"2024-09-10T20:21:22.150318+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:22.150318+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":458592720,"num_objects":55985,"num_object_clones":0,"num_object_copies":223940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52571,"num_write_kb":420473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14778368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":801415168,"data_stored":801415168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56042,"ondisk_log_size":56042,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400208,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93971632,"statfs":{"total":96636764160,"available":96226951168,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399700,"kb_used_data":224124,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93972140,"statfs":{"total":96636764160,"available":96227471360,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228560,"kb_used_data":110784,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94143280,"statfs":{"total":96636764160,"available":96402718720,"internally_reserved":0,"allocated":113442816,"data_stored":113257676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120581842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400148,"kb_used_data":224124,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93971692,"statfs":{"total":96636764160,"available":96227012608,"internally_reserved":0,"allocated":229502976,"data_stored":229317836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113766400,"data_stored":113766400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229216256,"data_stored":229216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:33.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 112 20 57125 57105 3.98278 2.5625 0.0352514 0.0392122 2024-09-10T20:21:34.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 113 20 57532 57512 3.97567 3.17969 0.0604507 0.0392873 2024-09-10T20:21:35.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 114 20 57930 57910 3.96807 3.10938 0.0585322 0.039362 2024-09-10T20:21:35.871 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:36.466 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:36.467 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:36.494 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-10T20:21:35.915863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468185552,"num_objects":57156,"num_object_clones":0,"num_object_copies":228624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53742,"num_write_kb":429841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":17424384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57213,"ondisk_log_size":57213,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1462944,"kb_used_data":800864,"kb_used_omap":14,"kb_used_meta":662001,"kb_avail":376024416,"statfs":{"total":386547056640,"available":385049001984,"internally_reserved":0,"allocated":820084736,"data_stored":819307244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":677889590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,12,5,14,11,11,3,0,0,0,0,0,0,20],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9592832,"num_objects":1171,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":9368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1540,"num_bytes_recovered":12599296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001769"},"pg_stats":[{"pgid":"1.f","version":"36'3578","reported_seq":3627,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.459191+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:34.459191+0000","last_peered":"2024-09-10T20:21:34.459191+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:34.459191+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3578,"log_dups_size":0,"ondisk_log_size":3578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":28624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3568","reported_seq":3670,"reported_epoch":36,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:21:32.759420+0000","last_change":"2024-09-10T20:21:31.215079+0000","last_active":"2024-09-10T20:21:32.759420+0000","last_peered":"2024-09-10T20:21:32.759420+0000","last_clean":"2024-09-10T20:21:32.759420+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:32.759420+0000","last_undegraded":"2024-09-10T20:21:32.759420+0000","last_fullsized":"2024-09-10T20:21:32.759420+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":2349,"log_size":3568,"log_dups_size":0,"ondisk_log_size":3568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29193376,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3566,"num_write_kb":28524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1995,"num_bytes_recovered":16310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3504","reported_seq":3588,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.900866+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:31.900866+0000","last_peered":"2024-09-10T20:21:31.900866+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:31.900866+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28678736,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":28014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3690","reported_seq":4034,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.195306+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:34.195306+0000","last_peered":"2024-09-10T20:21:34.195306+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:34.195306+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30192800,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1819}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3622","reported_seq":3681,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.502058+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:34.502058+0000","last_peered":"2024-09-10T20:21:34.502058+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:34.502058+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29643936,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3613","reported_seq":3662,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.506710+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:34.506710+0000","last_peered":"2024-09-10T20:21:34.506710+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:34.506710+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29597696,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1675}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3601","reported_seq":3674,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.241898+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:34.241898+0000","last_peered":"2024-09-10T20:21:34.241898+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:34.241898+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447328,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3643","reported_seq":3715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.188952+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:34.188952+0000","last_peered":"2024-09-10T20:21:34.188952+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:34.188952+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29799584,"num_objects":3638,"num_object_clones":0,"num_object_copies":14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":29116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3525","reported_seq":7486,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:34.549306+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:34.549306+0000","last_peered":"2024-09-10T20:21:34.549306+0000","last_clean":"2024-09-10T20:21:34.549306+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:34.549306+0000","last_undegraded":"2024-09-10T20:21:34.549306+0000","last_fullsized":"2024-09-10T20:21:34.549306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28785232,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3443","reported_seq":3527,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.912153+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:31.912153+0000","last_peered":"2024-09-10T20:21:31.912153+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:31.912153+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185760,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3441,"num_write_kb":27526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1538}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3560","reported_seq":3622,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.469781+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:34.469781+0000","last_peered":"2024-09-10T20:21:34.469781+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:34.469781+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29118192,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3614","reported_seq":3665,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.504068+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:34.504068+0000","last_peered":"2024-09-10T20:21:34.504068+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:34.504068+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29596240,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1635}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3456","reported_seq":3539,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.847813+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:31.847813+0000","last_peered":"2024-09-10T20:21:31.847813+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:31.847813+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301904,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3625","reported_seq":3683,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.506121+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:34.506121+0000","last_peered":"2024-09-10T20:21:34.506121+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:34.506121+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3625,"log_dups_size":0,"ondisk_log_size":3625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29660320,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3623,"num_write_kb":28980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1646}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3610","reported_seq":3697,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.914459+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:31.914459+0000","last_peered":"2024-09-10T20:21:31.914459+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:31.914459+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29545632,"num_objects":3607,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3608,"num_write_kb":28861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1645}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3561","reported_seq":3632,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.131742+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:34.131742+0000","last_peered":"2024-09-10T20:21:34.131742+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:34.131742+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127840,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468185552,"num_objects":57156,"num_object_clones":0,"num_object_copies":228624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53742,"num_write_kb":429841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":17424384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":822181888,"data_stored":822181888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57213,"ondisk_log_size":57213,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409568,"kb_used_data":229328,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93962272,"statfs":{"total":96636764160,"available":96217366528,"internally_reserved":0,"allocated":234831872,"data_stored":234637499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,13,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406304,"kb_used_data":226576,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93965536,"statfs":{"total":96636764160,"available":96220708864,"internally_reserved":0,"allocated":232013824,"data_stored":231819451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243076,"kb_used_data":116980,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94128764,"statfs":{"total":96636764160,"available":96387854336,"internally_reserved":0,"allocated":119787520,"data_stored":119593147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403996,"kb_used_data":227980,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93967844,"statfs":{"total":96636764160,"available":96223072256,"internally_reserved":0,"allocated":233451520,"data_stored":233257147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234778624,"data_stored":234778624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120377344,"data_stored":120377344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231755776,"data_stored":231755776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":235270144,"data_stored":235270144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:36.495 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:36.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 115 20 58276 58256 3.95707 2.70312 0.146563 0.0394695 2024-09-10T20:21:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:37.096 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:37.128 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-10T20:21:35.915863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468185552,"num_objects":57156,"num_object_clones":0,"num_object_copies":228624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53742,"num_write_kb":429841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":17424384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57213,"ondisk_log_size":57213,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1462944,"kb_used_data":800864,"kb_used_omap":14,"kb_used_meta":662001,"kb_avail":376024416,"statfs":{"total":386547056640,"available":385049001984,"internally_reserved":0,"allocated":820084736,"data_stored":819307244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":677889590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,12,5,14,11,11,3,0,0,0,0,0,0,20],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9592832,"num_objects":1171,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":9368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1540,"num_bytes_recovered":12599296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001769"},"pg_stats":[{"pgid":"1.f","version":"36'3578","reported_seq":3627,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.459191+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:34.459191+0000","last_peered":"2024-09-10T20:21:34.459191+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:34.459191+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3578,"log_dups_size":0,"ondisk_log_size":3578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":28624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3568","reported_seq":3670,"reported_epoch":36,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:21:32.759420+0000","last_change":"2024-09-10T20:21:31.215079+0000","last_active":"2024-09-10T20:21:32.759420+0000","last_peered":"2024-09-10T20:21:32.759420+0000","last_clean":"2024-09-10T20:21:32.759420+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:32.759420+0000","last_undegraded":"2024-09-10T20:21:32.759420+0000","last_fullsized":"2024-09-10T20:21:32.759420+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":2349,"log_size":3568,"log_dups_size":0,"ondisk_log_size":3568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29193376,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3566,"num_write_kb":28524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1995,"num_bytes_recovered":16310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3504","reported_seq":3588,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.900866+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:31.900866+0000","last_peered":"2024-09-10T20:21:31.900866+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:31.900866+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28678736,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":28014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3690","reported_seq":4034,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.195306+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:34.195306+0000","last_peered":"2024-09-10T20:21:34.195306+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:34.195306+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30192800,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1819}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3622","reported_seq":3681,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.502058+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:34.502058+0000","last_peered":"2024-09-10T20:21:34.502058+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:34.502058+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29643936,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3613","reported_seq":3662,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.506710+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:34.506710+0000","last_peered":"2024-09-10T20:21:34.506710+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:34.506710+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29597696,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1675}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3601","reported_seq":3674,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.241898+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:34.241898+0000","last_peered":"2024-09-10T20:21:34.241898+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:34.241898+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447328,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3643","reported_seq":3715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.188952+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:34.188952+0000","last_peered":"2024-09-10T20:21:34.188952+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:34.188952+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29799584,"num_objects":3638,"num_object_clones":0,"num_object_copies":14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":29116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3525","reported_seq":7486,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:34.549306+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:34.549306+0000","last_peered":"2024-09-10T20:21:34.549306+0000","last_clean":"2024-09-10T20:21:34.549306+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:34.549306+0000","last_undegraded":"2024-09-10T20:21:34.549306+0000","last_fullsized":"2024-09-10T20:21:34.549306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28785232,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3443","reported_seq":3527,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.912153+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:31.912153+0000","last_peered":"2024-09-10T20:21:31.912153+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:31.912153+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185760,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3441,"num_write_kb":27526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1538}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3560","reported_seq":3622,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.469781+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:34.469781+0000","last_peered":"2024-09-10T20:21:34.469781+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:34.469781+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29118192,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3614","reported_seq":3665,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.504068+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:34.504068+0000","last_peered":"2024-09-10T20:21:34.504068+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:34.504068+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29596240,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1635}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3456","reported_seq":3539,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.847813+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:31.847813+0000","last_peered":"2024-09-10T20:21:31.847813+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:31.847813+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301904,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3625","reported_seq":3683,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.506121+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:34.506121+0000","last_peered":"2024-09-10T20:21:34.506121+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:34.506121+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3625,"log_dups_size":0,"ondisk_log_size":3625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29660320,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3623,"num_write_kb":28980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1646}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3610","reported_seq":3697,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:31.914459+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:31.914459+0000","last_peered":"2024-09-10T20:21:31.914459+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:31.914459+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29545632,"num_objects":3607,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3608,"num_write_kb":28861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1645}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3561","reported_seq":3632,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.131742+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:34.131742+0000","last_peered":"2024-09-10T20:21:34.131742+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:34.131742+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127840,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468185552,"num_objects":57156,"num_object_clones":0,"num_object_copies":228624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53742,"num_write_kb":429841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":17424384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":822181888,"data_stored":822181888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57213,"ondisk_log_size":57213,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409568,"kb_used_data":229328,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93962272,"statfs":{"total":96636764160,"available":96217366528,"internally_reserved":0,"allocated":234831872,"data_stored":234637499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,13,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406304,"kb_used_data":226576,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93965536,"statfs":{"total":96636764160,"available":96220708864,"internally_reserved":0,"allocated":232013824,"data_stored":231819451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243076,"kb_used_data":116980,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94128764,"statfs":{"total":96636764160,"available":96387854336,"internally_reserved":0,"allocated":119787520,"data_stored":119593147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403996,"kb_used_data":227980,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93967844,"statfs":{"total":96636764160,"available":96223072256,"internally_reserved":0,"allocated":233451520,"data_stored":233257147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234778624,"data_stored":234778624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120377344,"data_stored":120377344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231755776,"data_stored":231755776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":235270144,"data_stored":235270144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:37.129 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:37.129 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'3578", 'reported_seq': 3627, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.459191+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:34.459191+0000', 'last_peered': '2024-09-10T20:21:34.459191+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:34.459191+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3578, 'log_dups_size': 0, 'ondisk_log_size': 3578, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29310976, 'num_objects': 3578, 'num_object_clones': 0, 'num_object_copies': 14312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3578, 'num_write_kb': 28624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1612}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:37.129 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-10T20:21:37.129 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'3568", 'reported_seq': 3670, 'reported_epoch': 36, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-10T20:21:32.759420+0000', 'last_change': '2024-09-10T20:21:31.215079+0000', 'last_active': '2024-09-10T20:21:32.759420+0000', 'last_peered': '2024-09-10T20:21:32.759420+0000', 'last_clean': '2024-09-10T20:21:32.759420+0000', 'last_became_active': '2024-09-10T20:21:31.068881+0000', 'last_became_peered': '2024-09-10T20:21:31.068881+0000', 'last_unstale': '2024-09-10T20:21:32.759420+0000', 'last_undegraded': '2024-09-10T20:21:32.759420+0000', 'last_fullsized': '2024-09-10T20:21:32.759420+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 36, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 2349, 'log_size': 3568, 'log_dups_size': 0, 'ondisk_log_size': 3568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 1s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29193376, 'num_objects': 3564, 'num_object_clones': 0, 'num_object_copies': 14256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3566, 'num_write_kb': 28524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1995, 'num_bytes_recovered': 16310272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [1, 0, 3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'3504", 'reported_seq': 3588, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:31.900866+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:31.900866+0000', 'last_peered': '2024-09-10T20:21:31.900866+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:31.900866+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3504, 'log_dups_size': 0, 'ondisk_log_size': 3504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28678736, 'num_objects': 3501, 'num_object_clones': 0, 'num_object_copies': 14004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3503, 'num_write_kb': 28014, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1539}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'3690", 'reported_seq': 4034, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.195306+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:21:34.195306+0000', 'last_peered': '2024-09-10T20:21:34.195306+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:34.195306+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3690, 'log_dups_size': 0, 'ondisk_log_size': 3690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30192800, 'num_objects': 3686, 'num_object_clones': 0, 'num_object_copies': 14744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1867, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3688, 'num_write_kb': 29493, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 138, 'num_bytes_recovered': 1114112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1819}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'3622", 'reported_seq': 3681, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.502058+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:34.502058+0000', 'last_peered': '2024-09-10T20:21:34.502058+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:34.502058+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3622, 'log_dups_size': 0, 'ondisk_log_size': 3622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29643936, 'num_objects': 3619, 'num_object_clones': 0, 'num_object_copies': 14476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3619, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3620, 'num_write_kb': 28957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1639}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'3613", 'reported_seq': 3662, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.506710+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:34.506710+0000', 'last_peered': '2024-09-10T20:21:34.506710+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:34.506710+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3613, 'log_dups_size': 0, 'ondisk_log_size': 3613, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29597696, 'num_objects': 3613, 'num_object_clones': 0, 'num_object_copies': 14452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3613, 'num_write_kb': 28904, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1675}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'3601", 'reported_seq': 3674, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.241898+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:34.241898+0000', 'last_peered': '2024-09-10T20:21:34.241898+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:34.241898+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3601, 'log_dups_size': 0, 'ondisk_log_size': 3601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29447328, 'num_objects': 3595, 'num_object_clones': 0, 'num_object_copies': 14380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3599, 'num_write_kb': 28772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1642}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:37.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'3643", 'reported_seq': 3715, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.188952+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:34.188952+0000', 'last_peered': '2024-09-10T20:21:34.188952+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:34.188952+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3643, 'log_dups_size': 0, 'ondisk_log_size': 3643, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29799584, 'num_objects': 3638, 'num_object_clones': 0, 'num_object_copies': 14552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3641, 'num_write_kb': 29116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1595}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'3443", 'reported_seq': 3527, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:31.912153+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:31.912153+0000', 'last_peered': '2024-09-10T20:21:31.912153+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:31.912153+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3443, 'log_dups_size': 0, 'ondisk_log_size': 3443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28185760, 'num_objects': 3441, 'num_object_clones': 0, 'num_object_copies': 13764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3441, 'num_write_kb': 27526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1538}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'3560", 'reported_seq': 3622, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.469781+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:34.469781+0000', 'last_peered': '2024-09-10T20:21:34.469781+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:34.469781+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3560, 'log_dups_size': 0, 'ondisk_log_size': 3560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29118192, 'num_objects': 3555, 'num_object_clones': 0, 'num_object_copies': 14220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3557, 'num_write_kb': 28444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'3614", 'reported_seq': 3665, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.504068+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:34.504068+0000', 'last_peered': '2024-09-10T20:21:34.504068+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:34.504068+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3614, 'log_dups_size': 0, 'ondisk_log_size': 3614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29596240, 'num_objects': 3613, 'num_object_clones': 0, 'num_object_copies': 14452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3613, 'num_write_kb': 28903, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1635}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'3456", 'reported_seq': 3539, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:31.847813+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:31.847813+0000', 'last_peered': '2024-09-10T20:21:31.847813+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:31.847813+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3456, 'log_dups_size': 0, 'ondisk_log_size': 3456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28301904, 'num_objects': 3455, 'num_object_clones': 0, 'num_object_copies': 13820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3455, 'num_write_kb': 27639, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:37.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'3625", 'reported_seq': 3683, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.506121+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:34.506121+0000', 'last_peered': '2024-09-10T20:21:34.506121+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:34.506121+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3625, 'log_dups_size': 0, 'ondisk_log_size': 3625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29660320, 'num_objects': 3621, 'num_object_clones': 0, 'num_object_copies': 14484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3623, 'num_write_kb': 28980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1646}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:37.132 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:37.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'3610", 'reported_seq': 3697, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:31.914459+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:31.914459+0000', 'last_peered': '2024-09-10T20:21:31.914459+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:31.914459+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3610, 'log_dups_size': 0, 'ondisk_log_size': 3610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29545632, 'num_objects': 3607, 'num_object_clones': 0, 'num_object_copies': 14428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3608, 'num_write_kb': 28861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1645}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:37.132 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:37.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'3561", 'reported_seq': 3632, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:34.131742+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:34.131742+0000', 'last_peered': '2024-09-10T20:21:34.131742+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:34.131742+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3561, 'log_dups_size': 0, 'ondisk_log_size': 3561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29127840, 'num_objects': 3556, 'num_object_clones': 0, 'num_object_copies': 14224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3559, 'num_write_kb': 28460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1577}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:37.132 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:37.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 116 20 58631 58611 3.94686 2.77344 0.0651469 0.0395769 2024-09-10T20:21:37.868 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:37.868 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27430/228624 objects degraded (11.998%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":186,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57156,"data_bytes":468185552,"bytes_used":1498054656,"bytes_avail":385049001984,"bytes_total":386547056640,"degraded_objects":27430,"degraded_total":228624,"degraded_ratio":0.11997865490937083,"recovering_objects_per_sec":192,"recovering_bytes_per_sec":1574563,"recovering_keys_per_sec":0,"num_objects_recovered":1540,"num_bytes_recovered":12599296,"num_keys_recovered":0,"write_bytes_sec":1198838,"read_op_per_sec":0,"write_op_per_sec":146},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:21:37.898 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:21:37.898 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:38.486 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:38.524 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-10T20:21:37.916451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472890672,"num_objects":57730,"num_object_clones":0,"num_object_copies":230920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50761,"num_write_kb":405997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57789,"ondisk_log_size":57789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1486236,"kb_used_data":815836,"kb_used_omap":14,"kb_used_meta":670321,"kb_avail":376001124,"statfs":{"total":386547056640,"available":385025150976,"internally_reserved":0,"allocated":835416064,"data_stored":834638572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":686409270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,12,8,14,12,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14297952,"num_objects":1745,"num_object_clones":0,"num_object_copies":6980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1810,"num_write_kb":-14476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1672,"num_bytes_recovered":-13664256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001969"},"pg_stats":[{"pgid":"1.f","version":"36'3578","reported_seq":3627,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.459191+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:34.459191+0000","last_peered":"2024-09-10T20:21:34.459191+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:34.459191+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3578,"log_dups_size":0,"ondisk_log_size":3578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":28624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3683","reported_seq":3804,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:37.784919+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:37.784919+0000","last_peered":"2024-09-10T20:21:37.784919+0000","last_clean":"2024-09-10T20:21:37.784919+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:37.784919+0000","last_undegraded":"2024-09-10T20:21:37.784919+0000","last_fullsized":"2024-09-10T20:21:37.784919+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30121984,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3627","reported_seq":3711,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.138130+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:37.138130+0000","last_peered":"2024-09-10T20:21:37.138130+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:37.138130+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29686352,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1662}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3690","reported_seq":4034,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.195306+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:34.195306+0000","last_peered":"2024-09-10T20:21:34.195306+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:34.195306+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30192800,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1819}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3622","reported_seq":3681,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.502058+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:34.502058+0000","last_peered":"2024-09-10T20:21:34.502058+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:34.502058+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29643936,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3613","reported_seq":3662,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.506710+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:34.506710+0000","last_peered":"2024-09-10T20:21:34.506710+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:34.506710+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29597696,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1675}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3601","reported_seq":3674,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.241898+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:34.241898+0000","last_peered":"2024-09-10T20:21:34.241898+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:34.241898+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447328,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3643","reported_seq":3715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.188952+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:34.188952+0000","last_peered":"2024-09-10T20:21:34.188952+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:34.188952+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29799584,"num_objects":3638,"num_object_clones":0,"num_object_copies":14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":29116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3525","reported_seq":7486,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:34.549306+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:34.549306+0000","last_peered":"2024-09-10T20:21:34.549306+0000","last_clean":"2024-09-10T20:21:34.549306+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:34.549306+0000","last_undegraded":"2024-09-10T20:21:34.549306+0000","last_fullsized":"2024-09-10T20:21:34.549306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28785232,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3571","reported_seq":3653,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.035049+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:37.035049+0000","last_peered":"2024-09-10T20:21:37.035049+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:37.035049+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3571,"log_dups_size":0,"ondisk_log_size":3571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29234336,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":28550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3560","reported_seq":3622,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.469781+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:34.469781+0000","last_peered":"2024-09-10T20:21:34.469781+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:34.469781+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29118192,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3614","reported_seq":3665,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.504068+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:34.504068+0000","last_peered":"2024-09-10T20:21:34.504068+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:34.504068+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29596240,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1635}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3544","reported_seq":3626,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.122971+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:37.122971+0000","last_peered":"2024-09-10T20:21:37.122971+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:37.122971+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29022800,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3543,"num_write_kb":28343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3625","reported_seq":3683,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.506121+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:34.506121+0000","last_peered":"2024-09-10T20:21:34.506121+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:34.506121+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3625,"log_dups_size":0,"ondisk_log_size":3625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29660320,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3623,"num_write_kb":28980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1646}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3732","reported_seq":3819,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.076529+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:37.076529+0000","last_peered":"2024-09-10T20:21:37.076529+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:37.076529+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30545056,"num_objects":3729,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3561","reported_seq":3632,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:34.131742+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:34.131742+0000","last_peered":"2024-09-10T20:21:34.131742+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:34.131742+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127840,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":28460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":472890672,"num_objects":57730,"num_object_clones":0,"num_object_copies":230920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50761,"num_write_kb":405997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":837541888,"data_stored":837541888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57789,"ondisk_log_size":57789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409568,"kb_used_data":229328,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93962272,"statfs":{"total":96636764160,"available":96217366528,"internally_reserved":0,"allocated":234831872,"data_stored":234637499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,13,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417596,"kb_used_data":233708,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93954244,"statfs":{"total":96636764160,"available":96209145856,"internally_reserved":0,"allocated":239316992,"data_stored":239122619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,3,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255076,"kb_used_data":124820,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94116764,"statfs":{"total":96636764160,"available":96375566336,"internally_reserved":0,"allocated":127815680,"data_stored":127621307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403996,"kb_used_data":227980,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93967844,"statfs":{"total":96636764160,"available":96223072256,"internally_reserved":0,"allocated":233451520,"data_stored":233257147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234778624,"data_stored":234778624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128249856,"data_stored":128249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":239243264,"data_stored":239243264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":235270144,"data_stored":235270144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:38.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 117 20 58998 58978 3.93763 2.86719 0.0327768 0.0396661 2024-09-10T20:21:39.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 118 20 59341 59321 3.92697 2.67969 0.0109215 0.0397262 2024-09-10T20:21:40.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 119 20 59739 59719 3.9201 3.10938 0.0509752 0.0398481 2024-09-10T20:21:41.527 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:41.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:21:41.783070+0000 min lat: 0.00230373 max lat: 12.2408 avg lat: 0.0399126 2024-09-10T20:21:41.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:21:41.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 120 20 60089 60069 3.91022 2.73438 0.0110239 0.0399126 2024-09-10T20:21:42.018 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:42.018 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:42.047 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-10T20:21:41.917532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483417392,"num_objects":59015,"num_object_clones":0,"num_object_copies":236060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52046,"num_write_kb":416277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59074,"ondisk_log_size":59074,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1549952,"kb_used_data":830972,"kb_used_omap":14,"kb_used_meta":718897,"kb_avail":375937408,"statfs":{"total":386547056640,"available":384959905792,"internally_reserved":0,"allocated":850915328,"data_stored":850137836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":736151094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,9,13,11,14,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24824672,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-525,"num_write_kb":-4196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1597,"num_bytes_recovered":-13049856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.884998"},"pg_stats":[{"pgid":"1.f","version":"36'3691","reported_seq":3739,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.538533+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:39.538533+0000","last_peered":"2024-09-10T20:21:39.538533+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:39.538533+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30236672,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":29528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3683","reported_seq":3804,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:37.784919+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:37.784919+0000","last_peered":"2024-09-10T20:21:37.784919+0000","last_clean":"2024-09-10T20:21:37.784919+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:37.784919+0000","last_undegraded":"2024-09-10T20:21:37.784919+0000","last_fullsized":"2024-09-10T20:21:37.784919+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30121984,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3627","reported_seq":3711,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.138130+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:37.138130+0000","last_peered":"2024-09-10T20:21:37.138130+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:37.138130+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29686352,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1662}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3789","reported_seq":4282,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.283101+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:39.283101+0000","last_peered":"2024-09-10T20:21:39.283101+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:39.283101+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31003808,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":30285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3743","reported_seq":3803,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.530216+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:39.530216+0000","last_peered":"2024-09-10T20:21:39.530216+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:39.530216+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30635168,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3741,"num_write_kb":29925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3737","reported_seq":3786,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.582239+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:39.582239+0000","last_peered":"2024-09-10T20:21:39.582239+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:39.582239+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30613504,"num_objects":3737,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":29896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3726","reported_seq":3799,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.245327+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:39.245327+0000","last_peered":"2024-09-10T20:21:39.245327+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:39.245327+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1767}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3762","reported_seq":3834,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.211058+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:39.211058+0000","last_peered":"2024-09-10T20:21:39.211058+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:39.211058+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774432,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3650","reported_seq":7612,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:39.477129+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:39.477129+0000","last_peered":"2024-09-10T20:21:39.477129+0000","last_clean":"2024-09-10T20:21:39.477129+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:39.477129+0000","last_undegraded":"2024-09-10T20:21:39.477129+0000","last_fullsized":"2024-09-10T20:21:39.477129+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29809232,"num_objects":3639,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":1624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3571","reported_seq":3653,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.035049+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:37.035049+0000","last_peered":"2024-09-10T20:21:37.035049+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:37.035049+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3571,"log_dups_size":0,"ondisk_log_size":3571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29234336,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":28550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3671","reported_seq":3734,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.600542+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:39.600542+0000","last_peered":"2024-09-10T20:21:39.600542+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:39.600542+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3671,"log_dups_size":0,"ondisk_log_size":3671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30027504,"num_objects":3666,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1710}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3735","reported_seq":3783,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.548598+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:39.548598+0000","last_peered":"2024-09-10T20:21:39.548598+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:39.548598+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30587472,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3734,"num_write_kb":29871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3544","reported_seq":3626,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.122971+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:37.122971+0000","last_peered":"2024-09-10T20:21:37.122971+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:37.122971+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29022800,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3543,"num_write_kb":28343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3724","reported_seq":3782,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.580791+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:39.580791+0000","last_peered":"2024-09-10T20:21:39.580791+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:39.580791+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3732","reported_seq":3819,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.076529+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:37.076529+0000","last_peered":"2024-09-10T20:21:37.076529+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:37.076529+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30545056,"num_objects":3729,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3689","reported_seq":3761,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.261412+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:39.261412+0000","last_peered":"2024-09-10T20:21:39.261412+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:39.261412+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30176416,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":29484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":483417392,"num_objects":59015,"num_object_clones":0,"num_object_copies":236060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52046,"num_write_kb":416277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":852963328,"data_stored":852963328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59074,"ondisk_log_size":59074,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439304,"kb_used_data":236852,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93932536,"statfs":{"total":96636764160,"available":96186916864,"internally_reserved":0,"allocated":242536448,"data_stored":242342075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":207285968},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417596,"kb_used_data":233708,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93954244,"statfs":{"total":96636764160,"available":96209145856,"internally_reserved":0,"allocated":239316992,"data_stored":239122619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,3,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255076,"kb_used_data":124820,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94116764,"statfs":{"total":96636764160,"available":96375566336,"internally_reserved":0,"allocated":127815680,"data_stored":127621307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437976,"kb_used_data":235592,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93933864,"statfs":{"total":96636764160,"available":96188276736,"internally_reserved":0,"allocated":241246208,"data_stored":241051835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,8,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242483200,"data_stored":242483200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128249856,"data_stored":128249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":239243264,"data_stored":239243264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242987008,"data_stored":242987008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:42.048 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:42.638 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:42.670 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-10T20:21:41.917532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483417392,"num_objects":59015,"num_object_clones":0,"num_object_copies":236060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52046,"num_write_kb":416277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59074,"ondisk_log_size":59074,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1549952,"kb_used_data":830972,"kb_used_omap":14,"kb_used_meta":718897,"kb_avail":375937408,"statfs":{"total":386547056640,"available":384959905792,"internally_reserved":0,"allocated":850915328,"data_stored":850137836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":736151094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,9,13,11,14,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24824672,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-525,"num_write_kb":-4196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1597,"num_bytes_recovered":-13049856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.884998"},"pg_stats":[{"pgid":"1.f","version":"36'3691","reported_seq":3739,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.538533+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:39.538533+0000","last_peered":"2024-09-10T20:21:39.538533+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:39.538533+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30236672,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":29528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3683","reported_seq":3804,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:37.784919+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:37.784919+0000","last_peered":"2024-09-10T20:21:37.784919+0000","last_clean":"2024-09-10T20:21:37.784919+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:37.784919+0000","last_undegraded":"2024-09-10T20:21:37.784919+0000","last_fullsized":"2024-09-10T20:21:37.784919+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30121984,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3627","reported_seq":3711,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.138130+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:37.138130+0000","last_peered":"2024-09-10T20:21:37.138130+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:37.138130+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29686352,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1662}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3789","reported_seq":4282,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.283101+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:39.283101+0000","last_peered":"2024-09-10T20:21:39.283101+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:39.283101+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31003808,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":30285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3743","reported_seq":3803,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.530216+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:39.530216+0000","last_peered":"2024-09-10T20:21:39.530216+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:39.530216+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30635168,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3741,"num_write_kb":29925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3737","reported_seq":3786,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.582239+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:39.582239+0000","last_peered":"2024-09-10T20:21:39.582239+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:39.582239+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30613504,"num_objects":3737,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":29896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3726","reported_seq":3799,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.245327+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:39.245327+0000","last_peered":"2024-09-10T20:21:39.245327+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:39.245327+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1767}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3762","reported_seq":3834,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.211058+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:39.211058+0000","last_peered":"2024-09-10T20:21:39.211058+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:39.211058+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774432,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3650","reported_seq":7612,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:39.477129+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:39.477129+0000","last_peered":"2024-09-10T20:21:39.477129+0000","last_clean":"2024-09-10T20:21:39.477129+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:39.477129+0000","last_undegraded":"2024-09-10T20:21:39.477129+0000","last_fullsized":"2024-09-10T20:21:39.477129+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29809232,"num_objects":3639,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":1624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3571","reported_seq":3653,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.035049+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:37.035049+0000","last_peered":"2024-09-10T20:21:37.035049+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:37.035049+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3571,"log_dups_size":0,"ondisk_log_size":3571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29234336,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":28550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3671","reported_seq":3734,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.600542+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:39.600542+0000","last_peered":"2024-09-10T20:21:39.600542+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:39.600542+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3671,"log_dups_size":0,"ondisk_log_size":3671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30027504,"num_objects":3666,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1710}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3735","reported_seq":3783,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.548598+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:39.548598+0000","last_peered":"2024-09-10T20:21:39.548598+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:39.548598+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30587472,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3734,"num_write_kb":29871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3544","reported_seq":3626,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.122971+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:37.122971+0000","last_peered":"2024-09-10T20:21:37.122971+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:37.122971+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29022800,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3543,"num_write_kb":28343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3724","reported_seq":3782,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.580791+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:39.580791+0000","last_peered":"2024-09-10T20:21:39.580791+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:39.580791+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3732","reported_seq":3819,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:37.076529+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:37.076529+0000","last_peered":"2024-09-10T20:21:37.076529+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:37.076529+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30545056,"num_objects":3729,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3689","reported_seq":3761,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.261412+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:39.261412+0000","last_peered":"2024-09-10T20:21:39.261412+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:39.261412+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30176416,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":29484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":483417392,"num_objects":59015,"num_object_clones":0,"num_object_copies":236060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52046,"num_write_kb":416277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":852963328,"data_stored":852963328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59074,"ondisk_log_size":59074,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439304,"kb_used_data":236852,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93932536,"statfs":{"total":96636764160,"available":96186916864,"internally_reserved":0,"allocated":242536448,"data_stored":242342075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":207285968},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417596,"kb_used_data":233708,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93954244,"statfs":{"total":96636764160,"available":96209145856,"internally_reserved":0,"allocated":239316992,"data_stored":239122619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,3,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255076,"kb_used_data":124820,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94116764,"statfs":{"total":96636764160,"available":96375566336,"internally_reserved":0,"allocated":127815680,"data_stored":127621307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437976,"kb_used_data":235592,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93933864,"statfs":{"total":96636764160,"available":96188276736,"internally_reserved":0,"allocated":241246208,"data_stored":241051835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,8,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242483200,"data_stored":242483200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128249856,"data_stored":128249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":239243264,"data_stored":239243264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242987008,"data_stored":242987008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:42.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:42.672 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'3691", 'reported_seq': 3739, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.538533+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:39.538533+0000', 'last_peered': '2024-09-10T20:21:39.538533+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:39.538533+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3691, 'log_dups_size': 0, 'ondisk_log_size': 3691, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30236672, 'num_objects': 3691, 'num_object_clones': 0, 'num_object_copies': 14764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3691, 'num_write_kb': 29528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1725}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:42.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:42.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'3627", 'reported_seq': 3711, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:37.138130+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:37.138130+0000', 'last_peered': '2024-09-10T20:21:37.138130+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:37.138130+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3627, 'log_dups_size': 0, 'ondisk_log_size': 3627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29686352, 'num_objects': 3624, 'num_object_clones': 0, 'num_object_copies': 14496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3626, 'num_write_kb': 28998, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1662}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:42.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:42.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'3789", 'reported_seq': 4282, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.283101+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:21:39.283101+0000', 'last_peered': '2024-09-10T20:21:39.283101+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:39.283101+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3789, 'log_dups_size': 0, 'ondisk_log_size': 3789, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31003808, 'num_objects': 3785, 'num_object_clones': 0, 'num_object_copies': 15140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1792, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3787, 'num_write_kb': 30285, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 213, 'num_bytes_recovered': 1728512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:42.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:42.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'3743", 'reported_seq': 3803, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.530216+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:39.530216+0000', 'last_peered': '2024-09-10T20:21:39.530216+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:39.530216+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3743, 'log_dups_size': 0, 'ondisk_log_size': 3743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30635168, 'num_objects': 3740, 'num_object_clones': 0, 'num_object_copies': 14960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3741, 'num_write_kb': 29925, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1760}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:42.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:42.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'3737", 'reported_seq': 3786, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.582239+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:39.582239+0000', 'last_peered': '2024-09-10T20:21:39.582239+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:39.582239+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3737, 'log_dups_size': 0, 'ondisk_log_size': 3737, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30613504, 'num_objects': 3737, 'num_object_clones': 0, 'num_object_copies': 14948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3737, 'num_write_kb': 29896, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1799}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:42.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:42.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'3726", 'reported_seq': 3799, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.245327+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:39.245327+0000', 'last_peered': '2024-09-10T20:21:39.245327+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:39.245327+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3726, 'log_dups_size': 0, 'ondisk_log_size': 3726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30471328, 'num_objects': 3720, 'num_object_clones': 0, 'num_object_copies': 14880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3724, 'num_write_kb': 29772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1767}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:42.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:42.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'3762", 'reported_seq': 3834, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.211058+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:39.211058+0000', 'last_peered': '2024-09-10T20:21:39.211058+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:39.211058+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3762, 'log_dups_size': 0, 'ondisk_log_size': 3762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30774432, 'num_objects': 3757, 'num_object_clones': 0, 'num_object_copies': 15028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3760, 'num_write_kb': 30068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1714}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:42.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:42.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'3571", 'reported_seq': 3653, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:37.035049+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:37.035049+0000', 'last_peered': '2024-09-10T20:21:37.035049+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:37.035049+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3571, 'log_dups_size': 0, 'ondisk_log_size': 3571, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29234336, 'num_objects': 3569, 'num_object_clones': 0, 'num_object_copies': 14276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3569, 'num_write_kb': 28550, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1666}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:42.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:42.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'3671", 'reported_seq': 3734, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.600542+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:39.600542+0000', 'last_peered': '2024-09-10T20:21:39.600542+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:39.600542+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3671, 'log_dups_size': 0, 'ondisk_log_size': 3671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30027504, 'num_objects': 3666, 'num_object_clones': 0, 'num_object_copies': 14664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3668, 'num_write_kb': 29332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1710}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:42.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:42.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'3735", 'reported_seq': 3783, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.548598+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:39.548598+0000', 'last_peered': '2024-09-10T20:21:39.548598+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:39.548598+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3735, 'log_dups_size': 0, 'ondisk_log_size': 3735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30587472, 'num_objects': 3734, 'num_object_clones': 0, 'num_object_copies': 14936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3734, 'num_write_kb': 29871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1756}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:42.677 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:42.677 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'3544", 'reported_seq': 3626, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:37.122971+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:37.122971+0000', 'last_peered': '2024-09-10T20:21:37.122971+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:37.122971+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3544, 'log_dups_size': 0, 'ondisk_log_size': 3544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29022800, 'num_objects': 3543, 'num_object_clones': 0, 'num_object_copies': 14172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3543, 'num_write_kb': 28343, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1595}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:42.677 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:42.677 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'3724", 'reported_seq': 3782, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.580791+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:39.580791+0000', 'last_peered': '2024-09-10T20:21:39.580791+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:39.580791+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3724, 'log_dups_size': 0, 'ondisk_log_size': 3724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30471328, 'num_objects': 3720, 'num_object_clones': 0, 'num_object_copies': 14880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3722, 'num_write_kb': 29772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1745}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:42.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:42.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'3732", 'reported_seq': 3819, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:37.076529+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:37.076529+0000', 'last_peered': '2024-09-10T20:21:37.076529+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:37.076529+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3732, 'log_dups_size': 0, 'ondisk_log_size': 3732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30545056, 'num_objects': 3729, 'num_object_clones': 0, 'num_object_copies': 14916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3730, 'num_write_kb': 29837, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:42.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:42.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'3689", 'reported_seq': 3761, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:39.261412+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:39.261412+0000', 'last_peered': '2024-09-10T20:21:39.261412+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:39.261412+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3689, 'log_dups_size': 0, 'ondisk_log_size': 3689, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30176416, 'num_objects': 3684, 'num_object_clones': 0, 'num_object_copies': 14736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3687, 'num_write_kb': 29484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:42.679 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:42.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 121 20 60414 60394 3.89888 2.53906 0.0609425 0.0400649 2024-09-10T20:21:43.391 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:43.391 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27355/236060 objects degraded (11.588%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":192,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59015,"data_bytes":483417392,"bytes_used":1587150848,"bytes_avail":384959905792,"bytes_total":386547056640,"degraded_objects":27355,"degraded_total":236060,"degraded_ratio":0.11588155553672795},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:21:43.409 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:21:43.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:43.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 122 20 60759 60739 3.88902 2.69531 0.0324544 0.0401633 2024-09-10T20:21:44.005 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:44.006 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:44.033 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-10T20:21:43.918057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":488168752,"num_objects":59595,"num_object_clones":0,"num_object_copies":238380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52626,"num_write_kb":420917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59654,"ondisk_log_size":59654,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1557988,"kb_used_data":845920,"kb_used_omap":14,"kb_used_meta":711985,"kb_avail":375929372,"statfs":{"total":386547056640,"available":384951676928,"internally_reserved":0,"allocated":866222080,"data_stored":865444588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":729073206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,5,9,21,17,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29576032,"num_objects":3610,"num_object_clones":0,"num_object_copies":14440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1597,"num_bytes_recovered":-13049856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.f","version":"36'3691","reported_seq":3739,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.538533+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:39.538533+0000","last_peered":"2024-09-10T20:21:39.538533+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:39.538533+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30236672,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":29528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3795","reported_seq":3917,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:42.633818+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:42.633818+0000","last_peered":"2024-09-10T20:21:42.633818+0000","last_clean":"2024-09-10T20:21:42.633818+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:42.633818+0000","last_undegraded":"2024-09-10T20:21:42.633818+0000","last_fullsized":"2024-09-10T20:21:42.633818+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":236,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3752","reported_seq":3836,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:42.129560+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:42.129560+0000","last_peered":"2024-09-10T20:21:42.129560+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:42.129560+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30710352,"num_objects":3749,"num_object_clones":0,"num_object_copies":14996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":29998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3789","reported_seq":4282,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.283101+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:39.283101+0000","last_peered":"2024-09-10T20:21:39.283101+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:39.283101+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31003808,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":30285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3743","reported_seq":3803,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.530216+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:39.530216+0000","last_peered":"2024-09-10T20:21:39.530216+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:39.530216+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30635168,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3741,"num_write_kb":29925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3737","reported_seq":3786,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.582239+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:39.582239+0000","last_peered":"2024-09-10T20:21:39.582239+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:39.582239+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30613504,"num_objects":3737,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":29896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3726","reported_seq":3799,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.245327+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:39.245327+0000","last_peered":"2024-09-10T20:21:39.245327+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:39.245327+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1767}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3762","reported_seq":3834,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.211058+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:39.211058+0000","last_peered":"2024-09-10T20:21:39.211058+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:39.211058+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774432,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3650","reported_seq":7612,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:39.477129+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:39.477129+0000","last_peered":"2024-09-10T20:21:39.477129+0000","last_clean":"2024-09-10T20:21:39.477129+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:39.477129+0000","last_undegraded":"2024-09-10T20:21:39.477129+0000","last_fullsized":"2024-09-10T20:21:39.477129+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29809232,"num_objects":3639,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":1624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3689","reported_seq":3774,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:42.068691+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:42.068691+0000","last_peered":"2024-09-10T20:21:42.068691+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:42.068691+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30200992,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":29494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1784}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3671","reported_seq":3734,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.600542+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:39.600542+0000","last_peered":"2024-09-10T20:21:39.600542+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:39.600542+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3671,"log_dups_size":0,"ondisk_log_size":3671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30027504,"num_objects":3666,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1710}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3735","reported_seq":3783,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.548598+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:39.548598+0000","last_peered":"2024-09-10T20:21:39.548598+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:39.548598+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30587472,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3734,"num_write_kb":29871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3655","reported_seq":3735,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:42.144267+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:42.144267+0000","last_peered":"2024-09-10T20:21:42.144267+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:42.144267+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29932112,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3724","reported_seq":3782,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.580791+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:39.580791+0000","last_peered":"2024-09-10T20:21:39.580791+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:39.580791+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3846","reported_seq":3933,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:42.058806+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:42.058806+0000","last_peered":"2024-09-10T20:21:42.058806+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:42.058806+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3846,"log_dups_size":0,"ondisk_log_size":3846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31478944,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":30749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3689","reported_seq":3761,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:39.261412+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:39.261412+0000","last_peered":"2024-09-10T20:21:39.261412+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:39.261412+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30176416,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":29484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":488168752,"num_objects":59595,"num_object_clones":0,"num_object_copies":238380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52626,"num_write_kb":420917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":868065280,"data_stored":868065280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59654,"ondisk_log_size":59654,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439304,"kb_used_data":236852,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93932536,"statfs":{"total":96636764160,"available":96186916864,"internally_reserved":0,"allocated":242536448,"data_stored":242342075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":207285968},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":430404,"kb_used_data":240948,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93941436,"statfs":{"total":96636764160,"available":96196030464,"internally_reserved":0,"allocated":246730752,"data_stored":246536379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,5,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":250304,"kb_used_data":132528,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94121536,"statfs":{"total":96636764160,"available":96380452864,"internally_reserved":0,"allocated":135708672,"data_stored":135514299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120581842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437976,"kb_used_data":235592,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93933864,"statfs":{"total":96636764160,"available":96188276736,"internally_reserved":0,"allocated":241246208,"data_stored":241051835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,8,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242483200,"data_stored":242483200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136036352,"data_stored":136036352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246558720,"data_stored":246558720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242987008,"data_stored":242987008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:44.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 123 20 61132 61112 3.88109 2.91406 0.0727014 0.0402468 2024-09-10T20:21:45.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 124 20 61515 61495 3.87392 2.99219 0.0510088 0.0403185 2024-09-10T20:21:46.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 125 20 61916 61896 3.86799 3.13281 0.0568444 0.040384 2024-09-10T20:21:47.035 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:47.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 126 20 62247 62227 3.85781 2.58594 0.00460171 0.0404585 2024-09-10T20:21:48.243 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-10T20:21:47.919232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502766896,"num_objects":61377,"num_object_clones":0,"num_object_copies":245508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54408,"num_write_kb":435173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61436,"ondisk_log_size":61436,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1641464,"kb_used_data":875256,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375845896,"statfs":{"total":386547056640,"available":384866197504,"internally_reserved":0,"allocated":896262144,"data_stored":895484652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":784516662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,6,5,12,19,15,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34581344,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":5332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1815,"num_bytes_recovered":-14835712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"1.f","version":"36'3815","reported_seq":3862,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.673231+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:44.673231+0000","last_peered":"2024-09-10T20:21:44.673231+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:44.673231+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31252480,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1849}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3905","reported_seq":4024,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:47.813659+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:47.813659+0000","last_peered":"2024-09-10T20:21:47.813659+0000","last_clean":"2024-09-10T20:21:47.813659+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:47.813659+0000","last_undegraded":"2024-09-10T20:21:47.813659+0000","last_fullsized":"2024-09-10T20:21:47.813659+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":346,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3883","reported_seq":3967,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.187586+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:47.187586+0000","last_peered":"2024-09-10T20:21:47.187586+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:47.187586+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783504,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1918}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3902","reported_seq":4606,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.304949+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:44.304949+0000","last_peered":"2024-09-10T20:21:44.304949+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:44.304949+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31929504,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3900,"num_write_kb":31189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3845","reported_seq":3905,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.541771+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:44.541771+0000","last_peered":"2024-09-10T20:21:44.541771+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:44.541771+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470752,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3847","reported_seq":3895,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.658405+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:44.658405+0000","last_peered":"2024-09-10T20:21:44.658405+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:44.658405+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31514624,"num_objects":3847,"num_object_clones":0,"num_object_copies":15388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":30776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3838","reported_seq":3911,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.189168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:44.189168+0000","last_peered":"2024-09-10T20:21:44.189168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:44.189168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31388832,"num_objects":3832,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3836,"num_write_kb":30668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1879}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3876","reported_seq":3948,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.255274+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:44.255274+0000","last_peered":"2024-09-10T20:21:44.255274+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:44.255274+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31708320,"num_objects":3871,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3874,"num_write_kb":30980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3751","reported_seq":7713,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:44.630768+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:44.630768+0000","last_peered":"2024-09-10T20:21:44.630768+0000","last_clean":"2024-09-10T20:21:44.630768+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:44.630768+0000","last_undegraded":"2024-09-10T20:21:44.630768+0000","last_fullsized":"2024-09-10T20:21:44.630768+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30636624,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3803","reported_seq":3885,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.189719+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:47.189719+0000","last_peered":"2024-09-10T20:21:47.189719+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:47.189719+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3803,"log_dups_size":0,"ondisk_log_size":3803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134880,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3787","reported_seq":3850,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.603662+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:44.603662+0000","last_peered":"2024-09-10T20:21:44.603662+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:44.603662+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30977776,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3839","reported_seq":3890,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.692039+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:44.692039+0000","last_peered":"2024-09-10T20:21:44.692039+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:44.692039+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3839,"log_dups_size":0,"ondisk_log_size":3839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31439440,"num_objects":3838,"num_object_clones":0,"num_object_copies":15352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3838,"num_write_kb":30703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3772","reported_seq":3855,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.181156+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:47.181156+0000","last_peered":"2024-09-10T20:21:47.181156+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:47.181156+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30890576,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":30167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3835","reported_seq":3893,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.657390+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:44.657390+0000","last_peered":"2024-09-10T20:21:44.657390+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:44.657390+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31380640,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":30660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3954","reported_seq":4041,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.156630+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:47.156630+0000","last_peered":"2024-09-10T20:21:47.156630+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:47.156630+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363680,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3784","reported_seq":3856,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.191604+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:44.191604+0000","last_peered":"2024-09-10T20:21:44.191604+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:44.191604+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30954656,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":30244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":502766896,"num_objects":61377,"num_object_clones":0,"num_object_copies":245508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54408,"num_write_kb":435173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898015232,"data_stored":898015232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61436,"ondisk_log_size":61436,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452004,"kb_used_data":243988,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93919836,"statfs":{"total":96636764160,"available":96173912064,"internally_reserved":0,"allocated":249843712,"data_stored":249649339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460140,"kb_used_data":248476,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93911700,"statfs":{"total":96636764160,"available":96165580800,"internally_reserved":0,"allocated":254439424,"data_stored":254245051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,9,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280092,"kb_used_data":140108,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94091748,"statfs":{"total":96636764160,"available":96349949952,"internally_reserved":0,"allocated":143470592,"data_stored":143276219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":143322834},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449228,"kb_used_data":242684,"kb_used_omap":4,"kb_used_meta":206523,"kb_avail":93922612,"statfs":{"total":96636764160,"available":96176754688,"internally_reserved":0,"allocated":248508416,"data_stored":248314043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":211480271},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,9,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249671680,"data_stored":249671680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143839232,"data_stored":143839232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254246912,"data_stored":254246912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250257408,"data_stored":250257408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:48.277 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:48.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 127 20 62611 62591 3.84982 2.84375 0.0833834 0.0405718 2024-09-10T20:21:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:48.873 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:48.895 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-10T20:21:47.919232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502766896,"num_objects":61377,"num_object_clones":0,"num_object_copies":245508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54408,"num_write_kb":435173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61436,"ondisk_log_size":61436,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1641464,"kb_used_data":875256,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375845896,"statfs":{"total":386547056640,"available":384866197504,"internally_reserved":0,"allocated":896262144,"data_stored":895484652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":784516662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,6,5,12,19,15,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34581344,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":5332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1815,"num_bytes_recovered":-14835712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"1.f","version":"36'3815","reported_seq":3862,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.673231+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:44.673231+0000","last_peered":"2024-09-10T20:21:44.673231+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:44.673231+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31252480,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1849}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3905","reported_seq":4024,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:47.813659+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:47.813659+0000","last_peered":"2024-09-10T20:21:47.813659+0000","last_clean":"2024-09-10T20:21:47.813659+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:47.813659+0000","last_undegraded":"2024-09-10T20:21:47.813659+0000","last_fullsized":"2024-09-10T20:21:47.813659+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":346,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3883","reported_seq":3967,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.187586+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:47.187586+0000","last_peered":"2024-09-10T20:21:47.187586+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:47.187586+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783504,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1918}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'3902","reported_seq":4606,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.304949+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:44.304949+0000","last_peered":"2024-09-10T20:21:44.304949+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:44.304949+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31929504,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3900,"num_write_kb":31189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3845","reported_seq":3905,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.541771+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:44.541771+0000","last_peered":"2024-09-10T20:21:44.541771+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:44.541771+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470752,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3847","reported_seq":3895,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.658405+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:44.658405+0000","last_peered":"2024-09-10T20:21:44.658405+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:44.658405+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31514624,"num_objects":3847,"num_object_clones":0,"num_object_copies":15388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":30776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3838","reported_seq":3911,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.189168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:44.189168+0000","last_peered":"2024-09-10T20:21:44.189168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:44.189168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31388832,"num_objects":3832,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3836,"num_write_kb":30668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1879}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3876","reported_seq":3948,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.255274+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:44.255274+0000","last_peered":"2024-09-10T20:21:44.255274+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:44.255274+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31708320,"num_objects":3871,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3874,"num_write_kb":30980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3751","reported_seq":7713,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:44.630768+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:44.630768+0000","last_peered":"2024-09-10T20:21:44.630768+0000","last_clean":"2024-09-10T20:21:44.630768+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:44.630768+0000","last_undegraded":"2024-09-10T20:21:44.630768+0000","last_fullsized":"2024-09-10T20:21:44.630768+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30636624,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3803","reported_seq":3885,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.189719+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:47.189719+0000","last_peered":"2024-09-10T20:21:47.189719+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:47.189719+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3803,"log_dups_size":0,"ondisk_log_size":3803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134880,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3787","reported_seq":3850,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.603662+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:44.603662+0000","last_peered":"2024-09-10T20:21:44.603662+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:44.603662+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30977776,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3839","reported_seq":3890,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.692039+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:44.692039+0000","last_peered":"2024-09-10T20:21:44.692039+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:44.692039+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3839,"log_dups_size":0,"ondisk_log_size":3839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31439440,"num_objects":3838,"num_object_clones":0,"num_object_copies":15352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3838,"num_write_kb":30703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3772","reported_seq":3855,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.181156+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:47.181156+0000","last_peered":"2024-09-10T20:21:47.181156+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:47.181156+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30890576,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":30167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3835","reported_seq":3893,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.657390+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:44.657390+0000","last_peered":"2024-09-10T20:21:44.657390+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:44.657390+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31380640,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":30660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3954","reported_seq":4041,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.156630+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:47.156630+0000","last_peered":"2024-09-10T20:21:47.156630+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:47.156630+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363680,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3784","reported_seq":3856,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:44.191604+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:44.191604+0000","last_peered":"2024-09-10T20:21:44.191604+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:44.191604+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30954656,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":30244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":502766896,"num_objects":61377,"num_object_clones":0,"num_object_copies":245508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54408,"num_write_kb":435173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898015232,"data_stored":898015232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61436,"ondisk_log_size":61436,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452004,"kb_used_data":243988,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93919836,"statfs":{"total":96636764160,"available":96173912064,"internally_reserved":0,"allocated":249843712,"data_stored":249649339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460140,"kb_used_data":248476,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93911700,"statfs":{"total":96636764160,"available":96165580800,"internally_reserved":0,"allocated":254439424,"data_stored":254245051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,9,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280092,"kb_used_data":140108,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94091748,"statfs":{"total":96636764160,"available":96349949952,"internally_reserved":0,"allocated":143470592,"data_stored":143276219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":143322834},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449228,"kb_used_data":242684,"kb_used_omap":4,"kb_used_meta":206523,"kb_avail":93922612,"statfs":{"total":96636764160,"available":96176754688,"internally_reserved":0,"allocated":248508416,"data_stored":248314043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":211480271},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,9,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249671680,"data_stored":249671680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143839232,"data_stored":143839232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254246912,"data_stored":254246912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250257408,"data_stored":250257408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:48.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'3815", 'reported_seq': 3862, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.673231+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:44.673231+0000', 'last_peered': '2024-09-10T20:21:44.673231+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:44.673231+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3815, 'log_dups_size': 0, 'ondisk_log_size': 3815, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31252480, 'num_objects': 3815, 'num_object_clones': 0, 'num_object_copies': 15260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3815, 'num_write_kb': 30520, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1849}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'3883", 'reported_seq': 3967, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:47.187586+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:47.187586+0000', 'last_peered': '2024-09-10T20:21:47.187586+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:47.187586+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3883, 'log_dups_size': 0, 'ondisk_log_size': 3883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31783504, 'num_objects': 3880, 'num_object_clones': 0, 'num_object_copies': 15520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3882, 'num_write_kb': 31046, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1918}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'3902", 'reported_seq': 4606, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.304949+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:21:44.304949+0000', 'last_peered': '2024-09-10T20:21:44.304949+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:44.304949+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3902, 'log_dups_size': 0, 'ondisk_log_size': 3902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31929504, 'num_objects': 3898, 'num_object_clones': 0, 'num_object_copies': 15592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1687, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3900, 'num_write_kb': 31189, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 318, 'num_bytes_recovered': 2588672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'3845", 'reported_seq': 3905, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.541771+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:44.541771+0000', 'last_peered': '2024-09-10T20:21:44.541771+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:44.541771+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31470752, 'num_objects': 3842, 'num_object_clones': 0, 'num_object_copies': 15368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3843, 'num_write_kb': 30741, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1862}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:48.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'3847", 'reported_seq': 3895, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.658405+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:44.658405+0000', 'last_peered': '2024-09-10T20:21:44.658405+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:44.658405+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3847, 'log_dups_size': 0, 'ondisk_log_size': 3847, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31514624, 'num_objects': 3847, 'num_object_clones': 0, 'num_object_copies': 15388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3847, 'num_write_kb': 30776, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'3838", 'reported_seq': 3911, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.189168+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:44.189168+0000', 'last_peered': '2024-09-10T20:21:44.189168+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:44.189168+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3838, 'log_dups_size': 0, 'ondisk_log_size': 3838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31388832, 'num_objects': 3832, 'num_object_clones': 0, 'num_object_copies': 15328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3836, 'num_write_kb': 30668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1879}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'3876", 'reported_seq': 3948, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.255274+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:44.255274+0000', 'last_peered': '2024-09-10T20:21:44.255274+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:44.255274+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3876, 'log_dups_size': 0, 'ondisk_log_size': 3876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31708320, 'num_objects': 3871, 'num_object_clones': 0, 'num_object_copies': 15484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3874, 'num_write_kb': 30980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'3803", 'reported_seq': 3885, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:47.189719+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:47.189719+0000', 'last_peered': '2024-09-10T20:21:47.189719+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:47.189719+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3803, 'log_dups_size': 0, 'ondisk_log_size': 3803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31134880, 'num_objects': 3801, 'num_object_clones': 0, 'num_object_copies': 15204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3801, 'num_write_kb': 30406, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1898}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'3787", 'reported_seq': 3850, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.603662+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:44.603662+0000', 'last_peered': '2024-09-10T20:21:44.603662+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:44.603662+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3787, 'log_dups_size': 0, 'ondisk_log_size': 3787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30977776, 'num_objects': 3782, 'num_object_clones': 0, 'num_object_copies': 15128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3784, 'num_write_kb': 30260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1826}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'3839", 'reported_seq': 3890, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.692039+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:44.692039+0000', 'last_peered': '2024-09-10T20:21:44.692039+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:44.692039+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3839, 'log_dups_size': 0, 'ondisk_log_size': 3839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31439440, 'num_objects': 3838, 'num_object_clones': 0, 'num_object_copies': 15352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3838, 'num_write_kb': 30703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:48.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'3772", 'reported_seq': 3855, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:47.181156+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:47.181156+0000', 'last_peered': '2024-09-10T20:21:47.181156+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:47.181156+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3772, 'log_dups_size': 0, 'ondisk_log_size': 3772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30890576, 'num_objects': 3771, 'num_object_clones': 0, 'num_object_copies': 15084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3771, 'num_write_kb': 30167, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1823}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:48.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:48.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'3835", 'reported_seq': 3893, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.657390+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:44.657390+0000', 'last_peered': '2024-09-10T20:21:44.657390+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:44.657390+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3835, 'log_dups_size': 0, 'ondisk_log_size': 3835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31380640, 'num_objects': 3831, 'num_object_clones': 0, 'num_object_copies': 15324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3833, 'num_write_kb': 30660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1856}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:48.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:48.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'3954", 'reported_seq': 4041, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:47.156630+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:47.156630+0000', 'last_peered': '2024-09-10T20:21:47.156630+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:47.156630+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32363680, 'num_objects': 3951, 'num_object_clones': 0, 'num_object_copies': 15804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3952, 'num_write_kb': 31613, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:48.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:48.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'3784", 'reported_seq': 3856, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:44.191604+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:44.191604+0000', 'last_peered': '2024-09-10T20:21:44.191604+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:44.191604+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3784, 'log_dups_size': 0, 'ondisk_log_size': 3784, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30954656, 'num_objects': 3779, 'num_object_clones': 0, 'num_object_copies': 15116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3779, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3782, 'num_write_kb': 30244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1800}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:48.899 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:49.646 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27250/245508 objects degraded (11.099%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":198,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61377,"data_bytes":502766896,"bytes_used":1680859136,"bytes_avail":384866197504,"bytes_total":386547056640,"degraded_objects":27250,"degraded_total":245508,"degraded_ratio":0.11099434641641005,"write_bytes_sec":454869,"read_op_per_sec":0,"write_op_per_sec":55},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:21:49.671 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:21:49.671 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:49.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 128 20 62942 62922 3.83995 2.58594 0.0885683 0.0406757 2024-09-10T20:21:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:50.134 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:50.163 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-10T20:21:49.919771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513162544,"num_objects":62646,"num_object_clones":0,"num_object_copies":250584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55677,"num_write_kb":445325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62705,"ondisk_log_size":62705,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1665964,"kb_used_data":890028,"kb_used_omap":14,"kb_used_meta":775857,"kb_avail":375821396,"statfs":{"total":386547056640,"available":384841109504,"internally_reserved":0,"allocated":911388672,"data_stored":910611180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":794478134},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,9,21,21,16,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40271872,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":39328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003319"},"pg_stats":[{"pgid":"1.f","version":"36'3937","reported_seq":3987,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.770892+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:49.770892+0000","last_peered":"2024-09-10T20:21:49.770892+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:49.770892+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":31496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3905","reported_seq":4024,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:47.813659+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:47.813659+0000","last_peered":"2024-09-10T20:21:47.813659+0000","last_clean":"2024-09-10T20:21:47.813659+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:47.813659+0000","last_undegraded":"2024-09-10T20:21:47.813659+0000","last_fullsized":"2024-09-10T20:21:47.813659+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":346,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3883","reported_seq":3967,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.187586+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:47.187586+0000","last_peered":"2024-09-10T20:21:47.187586+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:47.187586+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783504,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1918}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4031","reported_seq":4857,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.314555+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:49.314555+0000","last_peered":"2024-09-10T20:21:49.314555+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:49.314555+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32986272,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":32221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3972","reported_seq":4032,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.566298+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:49.566298+0000","last_peered":"2024-09-10T20:21:49.566298+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:49.566298+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511136,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3957","reported_seq":4007,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.670664+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:49.670664+0000","last_peered":"2024-09-10T20:21:49.670664+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:49.670664+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32415744,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3957,"num_write_kb":31656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3944","reported_seq":4017,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.109743+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:49.109743+0000","last_peered":"2024-09-10T20:21:49.109743+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:49.109743+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3944,"log_dups_size":0,"ondisk_log_size":3944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257184,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":31516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1985}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3989","reported_seq":4061,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281229+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:49.281229+0000","last_peered":"2024-09-10T20:21:49.281229+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:49.281229+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634016,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3873","reported_seq":7835,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:49.586391+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:49.586391+0000","last_peered":"2024-09-10T20:21:49.586391+0000","last_clean":"2024-09-10T20:21:49.586391+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:49.586391+0000","last_undegraded":"2024-09-10T20:21:49.586391+0000","last_fullsized":"2024-09-10T20:21:49.586391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31636048,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":426,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3803","reported_seq":3885,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.189719+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:47.189719+0000","last_peered":"2024-09-10T20:21:47.189719+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:47.189719+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3803,"log_dups_size":0,"ondisk_log_size":3803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134880,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3902","reported_seq":3965,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.749250+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:49.749250+0000","last_peered":"2024-09-10T20:21:49.749250+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:49.749250+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31919856,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3899,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3941","reported_seq":3991,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.763310+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:49.763310+0000","last_peered":"2024-09-10T20:21:49.763310+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:49.763310+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275024,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":31519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1962}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3772","reported_seq":3855,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.181156+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:47.181156+0000","last_peered":"2024-09-10T20:21:47.181156+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:47.181156+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30890576,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":30167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3951","reported_seq":4010,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.689866+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:49.689866+0000","last_peered":"2024-09-10T20:21:49.689866+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:49.689866+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3951,"log_dups_size":0,"ondisk_log_size":3951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330912,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3949,"num_write_kb":31588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1972}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3954","reported_seq":4041,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.156630+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:47.156630+0000","last_peered":"2024-09-10T20:21:47.156630+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:47.156630+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363680,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3891","reported_seq":3963,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281540+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:49.281540+0000","last_peered":"2024-09-10T20:21:49.281540+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:49.281540+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3891,"log_dups_size":0,"ondisk_log_size":3891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31831200,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513162544,"num_objects":62646,"num_object_clones":0,"num_object_copies":250584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55677,"num_write_kb":445325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":913117184,"data_stored":913117184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62705,"ondisk_log_size":62705,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463468,"kb_used_data":251292,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93908372,"statfs":{"total":96636764160,"available":96162172928,"internally_reserved":0,"allocated":257323008,"data_stored":257128635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460140,"kb_used_data":248476,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93911700,"statfs":{"total":96636764160,"available":96165580800,"internally_reserved":0,"allocated":254439424,"data_stored":254245051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,9,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280092,"kb_used_data":140108,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94091748,"statfs":{"total":96636764160,"available":96349949952,"internally_reserved":0,"allocated":143470592,"data_stored":143276219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":143322834},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462264,"kb_used_data":250152,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93909576,"statfs":{"total":96636764160,"available":96163405824,"internally_reserved":0,"allocated":256155648,"data_stored":255961275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,9,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257224704,"data_stored":257224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143839232,"data_stored":143839232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254246912,"data_stored":254246912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257806336,"data_stored":257806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:50.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 129 20 63196 63176 3.82556 1.98438 0.0608094 0.0408332 2024-09-10T20:21:51.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 130 20 63574 63554 3.81885 2.95312 0.0429435 0.0409008 2024-09-10T20:21:52.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 131 20 63960 63940 3.81272 3.01562 0.0407616 0.0409696 2024-09-10T20:21:53.165 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:53.756 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:53.767 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-10T20:21:51.920147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513162544,"num_objects":62646,"num_object_clones":0,"num_object_copies":250584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55677,"num_write_kb":445325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62705,"ondisk_log_size":62705,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1665964,"kb_used_data":890028,"kb_used_omap":14,"kb_used_meta":775857,"kb_avail":375821396,"statfs":{"total":386547056640,"available":384841109504,"internally_reserved":0,"allocated":911388672,"data_stored":910611180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":794478134},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,9,21,21,16,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29745152,"num_objects":3631,"num_object_clones":0,"num_object_copies":14524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":29048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003141"},"pg_stats":[{"pgid":"1.f","version":"36'3937","reported_seq":3987,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.770892+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:49.770892+0000","last_peered":"2024-09-10T20:21:49.770892+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:49.770892+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":31496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3905","reported_seq":4024,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:47.813659+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:47.813659+0000","last_peered":"2024-09-10T20:21:47.813659+0000","last_clean":"2024-09-10T20:21:47.813659+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:47.813659+0000","last_undegraded":"2024-09-10T20:21:47.813659+0000","last_fullsized":"2024-09-10T20:21:47.813659+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":346,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3883","reported_seq":3967,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.187586+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:47.187586+0000","last_peered":"2024-09-10T20:21:47.187586+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:47.187586+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783504,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1918}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4031","reported_seq":4857,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.314555+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:49.314555+0000","last_peered":"2024-09-10T20:21:49.314555+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:49.314555+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32986272,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":32221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3972","reported_seq":4032,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.566298+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:49.566298+0000","last_peered":"2024-09-10T20:21:49.566298+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:49.566298+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511136,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3957","reported_seq":4007,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.670664+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:49.670664+0000","last_peered":"2024-09-10T20:21:49.670664+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:49.670664+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32415744,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3957,"num_write_kb":31656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3944","reported_seq":4017,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.109743+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:49.109743+0000","last_peered":"2024-09-10T20:21:49.109743+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:49.109743+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3944,"log_dups_size":0,"ondisk_log_size":3944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257184,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":31516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1985}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3989","reported_seq":4061,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281229+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:49.281229+0000","last_peered":"2024-09-10T20:21:49.281229+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:49.281229+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634016,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3873","reported_seq":7835,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:49.586391+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:49.586391+0000","last_peered":"2024-09-10T20:21:49.586391+0000","last_clean":"2024-09-10T20:21:49.586391+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:49.586391+0000","last_undegraded":"2024-09-10T20:21:49.586391+0000","last_fullsized":"2024-09-10T20:21:49.586391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31636048,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":426,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3803","reported_seq":3885,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.189719+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:47.189719+0000","last_peered":"2024-09-10T20:21:47.189719+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:47.189719+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3803,"log_dups_size":0,"ondisk_log_size":3803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134880,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3902","reported_seq":3965,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.749250+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:49.749250+0000","last_peered":"2024-09-10T20:21:49.749250+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:49.749250+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31919856,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3899,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3941","reported_seq":3991,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.763310+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:49.763310+0000","last_peered":"2024-09-10T20:21:49.763310+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:49.763310+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275024,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":31519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1962}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3772","reported_seq":3855,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.181156+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:47.181156+0000","last_peered":"2024-09-10T20:21:47.181156+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:47.181156+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30890576,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":30167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3951","reported_seq":4010,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.689866+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:49.689866+0000","last_peered":"2024-09-10T20:21:49.689866+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:49.689866+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3951,"log_dups_size":0,"ondisk_log_size":3951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330912,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3949,"num_write_kb":31588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1972}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'3954","reported_seq":4041,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:47.156630+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:47.156630+0000","last_peered":"2024-09-10T20:21:47.156630+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:47.156630+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363680,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3891","reported_seq":3963,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281540+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:49.281540+0000","last_peered":"2024-09-10T20:21:49.281540+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:49.281540+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3891,"log_dups_size":0,"ondisk_log_size":3891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31831200,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513162544,"num_objects":62646,"num_object_clones":0,"num_object_copies":250584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55677,"num_write_kb":445325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":913117184,"data_stored":913117184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62705,"ondisk_log_size":62705,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463468,"kb_used_data":251292,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93908372,"statfs":{"total":96636764160,"available":96162172928,"internally_reserved":0,"allocated":257323008,"data_stored":257128635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460140,"kb_used_data":248476,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93911700,"statfs":{"total":96636764160,"available":96165580800,"internally_reserved":0,"allocated":254439424,"data_stored":254245051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,9,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280092,"kb_used_data":140108,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94091748,"statfs":{"total":96636764160,"available":96349949952,"internally_reserved":0,"allocated":143470592,"data_stored":143276219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":143322834},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462264,"kb_used_data":250152,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93909576,"statfs":{"total":96636764160,"available":96163405824,"internally_reserved":0,"allocated":256155648,"data_stored":255961275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,9,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257224704,"data_stored":257224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143839232,"data_stored":143839232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254246912,"data_stored":254246912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257806336,"data_stored":257806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:53.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:53.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 132 20 64301 64281 3.80401 2.66406 0.0166363 0.0410386 2024-09-10T20:21:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:54.277 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:54.302 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-10T20:21:53.920631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517496112,"num_objects":63175,"num_object_clones":0,"num_object_copies":252700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56206,"num_write_kb":449557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63234,"ondisk_log_size":63234,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1706452,"kb_used_data":904148,"kb_used_omap":14,"kb_used_meta":802225,"kb_avail":375780908,"statfs":{"total":386547056640,"available":384799649792,"internally_reserved":0,"allocated":925847552,"data_stored":925070060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":821478966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,12,19,18,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34078720,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":33280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003099"},"pg_stats":[{"pgid":"1.f","version":"36'3937","reported_seq":3987,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.770892+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:49.770892+0000","last_peered":"2024-09-10T20:21:49.770892+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:49.770892+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":31496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4017","reported_seq":4139,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:52.863568+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:52.863568+0000","last_peered":"2024-09-10T20:21:52.863568+0000","last_clean":"2024-09-10T20:21:52.863568+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:52.863568+0000","last_undegraded":"2024-09-10T20:21:52.863568+0000","last_fullsized":"2024-09-10T20:21:52.863568+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32858112,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":458,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3981","reported_seq":4065,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.041796+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:52.041796+0000","last_peered":"2024-09-10T20:21:52.041796+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:52.041796+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32586320,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4031","reported_seq":4857,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.314555+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:49.314555+0000","last_peered":"2024-09-10T20:21:49.314555+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:49.314555+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32986272,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":32221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3972","reported_seq":4032,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.566298+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:49.566298+0000","last_peered":"2024-09-10T20:21:49.566298+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:49.566298+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511136,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3957","reported_seq":4007,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.670664+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:49.670664+0000","last_peered":"2024-09-10T20:21:49.670664+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:49.670664+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32415744,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3957,"num_write_kb":31656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3944","reported_seq":4017,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.109743+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:49.109743+0000","last_peered":"2024-09-10T20:21:49.109743+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:49.109743+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3944,"log_dups_size":0,"ondisk_log_size":3944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257184,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":31516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1985}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3989","reported_seq":4061,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281229+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:49.281229+0000","last_peered":"2024-09-10T20:21:49.281229+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:49.281229+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634016,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3873","reported_seq":7835,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:49.586391+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:49.586391+0000","last_peered":"2024-09-10T20:21:49.586391+0000","last_clean":"2024-09-10T20:21:49.586391+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:49.586391+0000","last_undegraded":"2024-09-10T20:21:49.586391+0000","last_fullsized":"2024-09-10T20:21:49.586391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31636048,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":426,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3910","reported_seq":3995,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.189881+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:52.189881+0000","last_peered":"2024-09-10T20:21:52.189881+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:52.189881+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32011424,"num_objects":3908,"num_object_clones":0,"num_object_copies":15632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3908,"num_write_kb":31262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2005}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3902","reported_seq":3965,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.749250+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:49.749250+0000","last_peered":"2024-09-10T20:21:49.749250+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:49.749250+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31919856,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3899,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3941","reported_seq":3991,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.763310+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:49.763310+0000","last_peered":"2024-09-10T20:21:49.763310+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:49.763310+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275024,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":31519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1962}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3850","reported_seq":3933,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.153734+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:52.153734+0000","last_peered":"2024-09-10T20:21:52.153734+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:52.153734+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529552,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3849,"num_write_kb":30791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3951","reported_seq":4010,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.689866+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:49.689866+0000","last_peered":"2024-09-10T20:21:49.689866+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:49.689866+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3951,"log_dups_size":0,"ondisk_log_size":3951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330912,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3949,"num_write_kb":31588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1972}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4088","reported_seq":4175,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.161937+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:52.161937+0000","last_peered":"2024-09-10T20:21:52.161937+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:52.161937+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461408,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":32685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2123}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3891","reported_seq":3963,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281540+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:49.281540+0000","last_peered":"2024-09-10T20:21:49.281540+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:49.281540+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3891,"log_dups_size":0,"ondisk_log_size":3891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31831200,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517496112,"num_objects":63175,"num_object_clones":0,"num_object_copies":252700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56206,"num_write_kb":449557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":927576064,"data_stored":927576064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63234,"ondisk_log_size":63234,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463468,"kb_used_data":251292,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93908372,"statfs":{"total":96636764160,"available":96162172928,"internally_reserved":0,"allocated":257323008,"data_stored":257128635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471100,"kb_used_data":255276,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93900740,"statfs":{"total":96636764160,"available":96154357760,"internally_reserved":0,"allocated":261402624,"data_stored":261208251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,8,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309620,"kb_used_data":147428,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94062220,"statfs":{"total":96636764160,"available":96319713280,"internally_reserved":0,"allocated":150966272,"data_stored":150771899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462264,"kb_used_data":250152,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93909576,"statfs":{"total":96636764160,"available":96163405824,"internally_reserved":0,"allocated":256155648,"data_stored":255961275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,9,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257224704,"data_stored":257224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151420928,"data_stored":151420928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261124096,"data_stored":261124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257806336,"data_stored":257806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:54.303 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:54.303 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'3937", 'reported_seq': 3987, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.770892+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:49.770892+0000', 'last_peered': '2024-09-10T20:21:49.770892+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:49.770892+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3937, 'log_dups_size': 0, 'ondisk_log_size': 3937, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32251904, 'num_objects': 3937, 'num_object_clones': 0, 'num_object_copies': 15748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3937, 'num_write_kb': 31496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1971}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:54.303 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:54.303 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'3981", 'reported_seq': 4065, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:52.041796+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:52.041796+0000', 'last_peered': '2024-09-10T20:21:52.041796+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:52.041796+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3981, 'log_dups_size': 0, 'ondisk_log_size': 3981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32586320, 'num_objects': 3978, 'num_object_clones': 0, 'num_object_copies': 15912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3980, 'num_write_kb': 31830, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2016}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4031", 'reported_seq': 4857, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.314555+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:21:49.314555+0000', 'last_peered': '2024-09-10T20:21:49.314555+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:49.314555+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4031, 'log_dups_size': 0, 'ondisk_log_size': 4031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32986272, 'num_objects': 4027, 'num_object_clones': 0, 'num_object_copies': 16108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1626, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4029, 'num_write_kb': 32221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 379, 'num_bytes_recovered': 3088384, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2401}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'3972", 'reported_seq': 4032, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.566298+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:49.566298+0000', 'last_peered': '2024-09-10T20:21:49.566298+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:49.566298+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3972, 'log_dups_size': 0, 'ondisk_log_size': 3972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32511136, 'num_objects': 3969, 'num_object_clones': 0, 'num_object_copies': 15876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3970, 'num_write_kb': 31757, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1989}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'3957", 'reported_seq': 4007, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.670664+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:49.670664+0000', 'last_peered': '2024-09-10T20:21:49.670664+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:49.670664+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3957, 'log_dups_size': 0, 'ondisk_log_size': 3957, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32415744, 'num_objects': 3957, 'num_object_clones': 0, 'num_object_copies': 15828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3957, 'num_write_kb': 31656, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2019}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'3944", 'reported_seq': 4017, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.109743+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:49.109743+0000', 'last_peered': '2024-09-10T20:21:49.109743+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:49.109743+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3944, 'log_dups_size': 0, 'ondisk_log_size': 3944, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32257184, 'num_objects': 3938, 'num_object_clones': 0, 'num_object_copies': 15752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3942, 'num_write_kb': 31516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1985}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'3989", 'reported_seq': 4061, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.281229+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:49.281229+0000', 'last_peered': '2024-09-10T20:21:49.281229+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:49.281229+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3989, 'log_dups_size': 0, 'ondisk_log_size': 3989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32634016, 'num_objects': 3984, 'num_object_clones': 0, 'num_object_copies': 15936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3987, 'num_write_kb': 31884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1941}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:54.304 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'3910", 'reported_seq': 3995, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:52.189881+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:52.189881+0000', 'last_peered': '2024-09-10T20:21:52.189881+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:52.189881+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3910, 'log_dups_size': 0, 'ondisk_log_size': 3910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32011424, 'num_objects': 3908, 'num_object_clones': 0, 'num_object_copies': 15632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3908, 'num_write_kb': 31262, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2005}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'3902", 'reported_seq': 3965, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.749250+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:49.749250+0000', 'last_peered': '2024-09-10T20:21:49.749250+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:49.749250+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3902, 'log_dups_size': 0, 'ondisk_log_size': 3902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31919856, 'num_objects': 3897, 'num_object_clones': 0, 'num_object_copies': 15588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3899, 'num_write_kb': 31180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1941}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'3941", 'reported_seq': 3991, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.763310+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:49.763310+0000', 'last_peered': '2024-09-10T20:21:49.763310+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:49.763310+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3941, 'log_dups_size': 0, 'ondisk_log_size': 3941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32275024, 'num_objects': 3940, 'num_object_clones': 0, 'num_object_copies': 15760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3940, 'num_write_kb': 31519, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'3850", 'reported_seq': 3933, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:52.153734+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:52.153734+0000', 'last_peered': '2024-09-10T20:21:52.153734+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:52.153734+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3850, 'log_dups_size': 0, 'ondisk_log_size': 3850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31529552, 'num_objects': 3849, 'num_object_clones': 0, 'num_object_copies': 15396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3849, 'num_write_kb': 30791, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'3951", 'reported_seq': 4010, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.689866+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:49.689866+0000', 'last_peered': '2024-09-10T20:21:49.689866+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:49.689866+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3951, 'log_dups_size': 0, 'ondisk_log_size': 3951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32330912, 'num_objects': 3947, 'num_object_clones': 0, 'num_object_copies': 15788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3949, 'num_write_kb': 31588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1972}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:54.305 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4088", 'reported_seq': 4175, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:52.161937+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:52.161937+0000', 'last_peered': '2024-09-10T20:21:52.161937+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:52.161937+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4088, 'log_dups_size': 0, 'ondisk_log_size': 4088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33461408, 'num_objects': 4085, 'num_object_clones': 0, 'num_object_copies': 16340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4086, 'num_write_kb': 32685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2123}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:54.306 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:54.306 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'3891", 'reported_seq': 3963, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:49.281540+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:49.281540+0000', 'last_peered': '2024-09-10T20:21:49.281540+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:49.281540+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3891, 'log_dups_size': 0, 'ondisk_log_size': 3891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31831200, 'num_objects': 3886, 'num_object_clones': 0, 'num_object_copies': 15544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3889, 'num_write_kb': 31100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:54.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:21:54.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 133 20 64617 64597 3.79397 2.46875 0.063023 0.0411703 2024-09-10T20:21:55.061 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27189/252700 objects degraded (10.759%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63175,"data_bytes":517496112,"bytes_used":1747406848,"bytes_avail":384799649792,"bytes_total":386547056640,"degraded_objects":27189,"degraded_total":252700,"degraded_ratio":0.10759398496240602,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":113293,"recovering_keys_per_sec":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"write_bytes_sec":2839159,"read_op_per_sec":0,"write_op_per_sec":346},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:21:55.089 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:21:55.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:55.689 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-10T20:21:53.920631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517496112,"num_objects":63175,"num_object_clones":0,"num_object_copies":252700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56206,"num_write_kb":449557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63234,"ondisk_log_size":63234,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1706452,"kb_used_data":904148,"kb_used_omap":14,"kb_used_meta":802225,"kb_avail":375780908,"statfs":{"total":386547056640,"available":384799649792,"internally_reserved":0,"allocated":925847552,"data_stored":925070060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":821478966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,12,19,18,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34078720,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":33280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003099"},"pg_stats":[{"pgid":"1.f","version":"36'3937","reported_seq":3987,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.770892+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:49.770892+0000","last_peered":"2024-09-10T20:21:49.770892+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:49.770892+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":31496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4017","reported_seq":4139,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:52.863568+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:52.863568+0000","last_peered":"2024-09-10T20:21:52.863568+0000","last_clean":"2024-09-10T20:21:52.863568+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:52.863568+0000","last_undegraded":"2024-09-10T20:21:52.863568+0000","last_fullsized":"2024-09-10T20:21:52.863568+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32858112,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":458,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3981","reported_seq":4065,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.041796+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:52.041796+0000","last_peered":"2024-09-10T20:21:52.041796+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:52.041796+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32586320,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4031","reported_seq":4857,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.314555+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:49.314555+0000","last_peered":"2024-09-10T20:21:49.314555+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:49.314555+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32986272,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":32221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'3972","reported_seq":4032,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.566298+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:49.566298+0000","last_peered":"2024-09-10T20:21:49.566298+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:49.566298+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511136,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'3957","reported_seq":4007,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.670664+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:49.670664+0000","last_peered":"2024-09-10T20:21:49.670664+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:49.670664+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32415744,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3957,"num_write_kb":31656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'3944","reported_seq":4017,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.109743+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:49.109743+0000","last_peered":"2024-09-10T20:21:49.109743+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:49.109743+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3944,"log_dups_size":0,"ondisk_log_size":3944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257184,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":31516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1985}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3989","reported_seq":4061,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281229+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:49.281229+0000","last_peered":"2024-09-10T20:21:49.281229+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:49.281229+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634016,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3873","reported_seq":7835,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:49.586391+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:49.586391+0000","last_peered":"2024-09-10T20:21:49.586391+0000","last_clean":"2024-09-10T20:21:49.586391+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:49.586391+0000","last_undegraded":"2024-09-10T20:21:49.586391+0000","last_fullsized":"2024-09-10T20:21:49.586391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31636048,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":426,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3910","reported_seq":3995,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.189881+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:52.189881+0000","last_peered":"2024-09-10T20:21:52.189881+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:52.189881+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32011424,"num_objects":3908,"num_object_clones":0,"num_object_copies":15632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3908,"num_write_kb":31262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2005}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3902","reported_seq":3965,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.749250+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:49.749250+0000","last_peered":"2024-09-10T20:21:49.749250+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:49.749250+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31919856,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3899,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'3941","reported_seq":3991,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.763310+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:49.763310+0000","last_peered":"2024-09-10T20:21:49.763310+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:49.763310+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275024,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":31519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1962}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3850","reported_seq":3933,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.153734+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:52.153734+0000","last_peered":"2024-09-10T20:21:52.153734+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:52.153734+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529552,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3849,"num_write_kb":30791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'3951","reported_seq":4010,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.689866+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:49.689866+0000","last_peered":"2024-09-10T20:21:49.689866+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:49.689866+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3951,"log_dups_size":0,"ondisk_log_size":3951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330912,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3949,"num_write_kb":31588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1972}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4088","reported_seq":4175,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:52.161937+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:52.161937+0000","last_peered":"2024-09-10T20:21:52.161937+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:52.161937+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461408,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":32685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2123}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'3891","reported_seq":3963,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:49.281540+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:49.281540+0000","last_peered":"2024-09-10T20:21:49.281540+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:49.281540+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3891,"log_dups_size":0,"ondisk_log_size":3891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31831200,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517496112,"num_objects":63175,"num_object_clones":0,"num_object_copies":252700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56206,"num_write_kb":449557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":927576064,"data_stored":927576064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63234,"ondisk_log_size":63234,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463468,"kb_used_data":251292,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93908372,"statfs":{"total":96636764160,"available":96162172928,"internally_reserved":0,"allocated":257323008,"data_stored":257128635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471100,"kb_used_data":255276,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93900740,"statfs":{"total":96636764160,"available":96154357760,"internally_reserved":0,"allocated":261402624,"data_stored":261208251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,8,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309620,"kb_used_data":147428,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94062220,"statfs":{"total":96636764160,"available":96319713280,"internally_reserved":0,"allocated":150966272,"data_stored":150771899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462264,"kb_used_data":250152,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93909576,"statfs":{"total":96636764160,"available":96163405824,"internally_reserved":0,"allocated":256155648,"data_stored":255961275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,9,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257224704,"data_stored":257224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151420928,"data_stored":151420928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261124096,"data_stored":261124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257806336,"data_stored":257806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:55.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 134 20 64960 64940 3.78566 2.67969 0.0372272 0.0412627 2024-09-10T20:21:56.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 135 20 65295 65275 3.777 2.61719 0.0363051 0.04136 2024-09-10T20:21:57.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 136 20 65613 65593 3.76749 2.48438 0.0715649 0.0414607 2024-09-10T20:21:58.722 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:58.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 137 20 65983 65963 3.76109 2.89062 0.114446 0.0415251 2024-09-10T20:21:59.299 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:59.300 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:59.329 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-10T20:21:57.921764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530373936,"num_objects":64747,"num_object_clones":0,"num_object_copies":258988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57778,"num_write_kb":462133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64806,"ondisk_log_size":64806,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1743484,"kb_used_data":924540,"kb_used_omap":14,"kb_used_meta":818865,"kb_avail":375743876,"statfs":{"total":386547056640,"available":384761729024,"internally_reserved":0,"allocated":946728960,"data_stored":945951468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":838518326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,3,3,10,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32358400,"num_objects":3950,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3950,"num_write_kb":31600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003124"},"pg_stats":[{"pgid":"1.f","version":"36'4043","reported_seq":4093,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.835246+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:54.835246+0000","last_peered":"2024-09-10T20:21:54.835246+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:54.835246+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33120256,"num_objects":4043,"num_object_clones":0,"num_object_copies":16172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4017","reported_seq":4139,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:52.863568+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:52.863568+0000","last_peered":"2024-09-10T20:21:52.863568+0000","last_clean":"2024-09-10T20:21:52.863568+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:52.863568+0000","last_undegraded":"2024-09-10T20:21:52.863568+0000","last_fullsized":"2024-09-10T20:21:52.863568+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32858112,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":458,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4099","reported_seq":4183,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:57.046918+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:57.046918+0000","last_peered":"2024-09-10T20:21:57.046918+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:57.046918+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33552976,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4126","reported_seq":5194,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.388539+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:54.388539+0000","last_peered":"2024-09-10T20:21:54.388539+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:54.388539+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4126,"log_dups_size":0,"ondisk_log_size":4126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33764512,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":32981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4064","reported_seq":4123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.756741+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:54.756741+0000","last_peered":"2024-09-10T20:21:54.756741+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:54.756741+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264800,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":32493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4073","reported_seq":4123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.611753+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:54.611753+0000","last_peered":"2024-09-10T20:21:54.611753+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:54.611753+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33366016,"num_objects":4073,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":32584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4056","reported_seq":4129,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.259693+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:54.259693+0000","last_peered":"2024-09-10T20:21:54.259693+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:54.259693+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33174688,"num_objects":4050,"num_object_clones":0,"num_object_copies":16200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":32412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4078","reported_seq":4150,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.161583+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:54.161583+0000","last_peered":"2024-09-10T20:21:54.161583+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:54.161583+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33363104,"num_objects":4073,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":32596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3992","reported_seq":7954,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:54.893708+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:54.893708+0000","last_peered":"2024-09-10T20:21:54.893708+0000","last_clean":"2024-09-10T20:21:54.893708+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:54.893708+0000","last_undegraded":"2024-09-10T20:21:54.893708+0000","last_fullsized":"2024-09-10T20:21:54.893708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32610896,"num_objects":3981,"num_object_clones":0,"num_object_copies":15924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":545,"num_write_kb":4360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4016","reported_seq":4101,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:57.098959+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:57.098959+0000","last_peered":"2024-09-10T20:21:57.098959+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:57.098959+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32879776,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":32110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3992","reported_seq":4055,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.667957+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:54.667957+0000","last_peered":"2024-09-10T20:21:54.667957+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:54.667957+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32657136,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":31900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4058","reported_seq":4107,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.849480+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:54.849480+0000","last_peered":"2024-09-10T20:21:54.849480+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:54.849480+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33233488,"num_objects":4057,"num_object_clones":0,"num_object_copies":16228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":32455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3943","reported_seq":4023,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:56.828502+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:56.828502+0000","last_peered":"2024-09-10T20:21:56.828502+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:56.828502+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32291408,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":31535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4047","reported_seq":4104,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.880091+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:54.880091+0000","last_peered":"2024-09-10T20:21:54.880091+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:54.880091+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33117344,"num_objects":4043,"num_object_clones":0,"num_object_copies":16172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4045,"num_write_kb":32356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2068}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4194","reported_seq":4281,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:56.898715+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:56.898715+0000","last_peered":"2024-09-10T20:21:56.898715+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:56.898715+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2229}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4008","reported_seq":4080,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.333905+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:54.333905+0000","last_peered":"2024-09-10T20:21:54.333905+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:54.333905+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32789664,"num_objects":4003,"num_object_clones":0,"num_object_copies":16012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4006,"num_write_kb":32036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530373936,"num_objects":64747,"num_object_clones":0,"num_object_copies":258988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57778,"num_write_kb":462133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":948305920,"data_stored":948305920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64806,"ondisk_log_size":64806,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478540,"kb_used_data":258044,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93893300,"statfs":{"total":96636764160,"available":96146739200,"internally_reserved":0,"allocated":264237056,"data_stored":264042683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481948,"kb_used_data":261964,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93889892,"statfs":{"total":96636764160,"available":96143249408,"internally_reserved":0,"allocated":268251136,"data_stored":268056763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309620,"kb_used_data":147428,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94062220,"statfs":{"total":96636764160,"available":96319713280,"internally_reserved":0,"allocated":150966272,"data_stored":150771899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473376,"kb_used_data":257104,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93898464,"statfs":{"total":96636764160,"available":96152027136,"internally_reserved":0,"allocated":263274496,"data_stored":263080123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264101888,"data_stored":264101888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151420928,"data_stored":151420928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267948032,"data_stored":267948032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":264835072,"data_stored":264835072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:59.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:21:59.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 138 20 66311 66291 3.75241 2.5625 0.1175 0.0416267 2024-09-10T20:21:59.908 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:21:59.909 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:21:59.935 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-10T20:21:57.921764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530373936,"num_objects":64747,"num_object_clones":0,"num_object_copies":258988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57778,"num_write_kb":462133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64806,"ondisk_log_size":64806,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1743484,"kb_used_data":924540,"kb_used_omap":14,"kb_used_meta":818865,"kb_avail":375743876,"statfs":{"total":386547056640,"available":384761729024,"internally_reserved":0,"allocated":946728960,"data_stored":945951468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":838518326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,3,3,10,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32358400,"num_objects":3950,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3950,"num_write_kb":31600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003124"},"pg_stats":[{"pgid":"1.f","version":"36'4043","reported_seq":4093,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.835246+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:54.835246+0000","last_peered":"2024-09-10T20:21:54.835246+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:54.835246+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33120256,"num_objects":4043,"num_object_clones":0,"num_object_copies":16172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4017","reported_seq":4139,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:52.863568+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:52.863568+0000","last_peered":"2024-09-10T20:21:52.863568+0000","last_clean":"2024-09-10T20:21:52.863568+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:52.863568+0000","last_undegraded":"2024-09-10T20:21:52.863568+0000","last_fullsized":"2024-09-10T20:21:52.863568+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32858112,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":458,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4099","reported_seq":4183,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:57.046918+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:57.046918+0000","last_peered":"2024-09-10T20:21:57.046918+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:57.046918+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33552976,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4126","reported_seq":5194,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.388539+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:54.388539+0000","last_peered":"2024-09-10T20:21:54.388539+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:54.388539+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4126,"log_dups_size":0,"ondisk_log_size":4126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33764512,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":32981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4064","reported_seq":4123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.756741+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:54.756741+0000","last_peered":"2024-09-10T20:21:54.756741+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:54.756741+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264800,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":32493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4073","reported_seq":4123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.611753+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:54.611753+0000","last_peered":"2024-09-10T20:21:54.611753+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:54.611753+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33366016,"num_objects":4073,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":32584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4056","reported_seq":4129,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.259693+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:54.259693+0000","last_peered":"2024-09-10T20:21:54.259693+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:54.259693+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33174688,"num_objects":4050,"num_object_clones":0,"num_object_copies":16200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":32412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4078","reported_seq":4150,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.161583+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:54.161583+0000","last_peered":"2024-09-10T20:21:54.161583+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:54.161583+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33363104,"num_objects":4073,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":32596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3992","reported_seq":7954,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:54.893708+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:54.893708+0000","last_peered":"2024-09-10T20:21:54.893708+0000","last_clean":"2024-09-10T20:21:54.893708+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:54.893708+0000","last_undegraded":"2024-09-10T20:21:54.893708+0000","last_fullsized":"2024-09-10T20:21:54.893708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32610896,"num_objects":3981,"num_object_clones":0,"num_object_copies":15924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":545,"num_write_kb":4360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4016","reported_seq":4101,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:57.098959+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:57.098959+0000","last_peered":"2024-09-10T20:21:57.098959+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:57.098959+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32879776,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":32110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3992","reported_seq":4055,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.667957+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:54.667957+0000","last_peered":"2024-09-10T20:21:54.667957+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:54.667957+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32657136,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":31900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4058","reported_seq":4107,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.849480+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:54.849480+0000","last_peered":"2024-09-10T20:21:54.849480+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:54.849480+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33233488,"num_objects":4057,"num_object_clones":0,"num_object_copies":16228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":32455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3943","reported_seq":4023,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:56.828502+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:56.828502+0000","last_peered":"2024-09-10T20:21:56.828502+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:56.828502+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32291408,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":31535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4047","reported_seq":4104,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.880091+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:54.880091+0000","last_peered":"2024-09-10T20:21:54.880091+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:54.880091+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33117344,"num_objects":4043,"num_object_clones":0,"num_object_copies":16172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4045,"num_write_kb":32356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2068}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4194","reported_seq":4281,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:56.898715+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:56.898715+0000","last_peered":"2024-09-10T20:21:56.898715+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:56.898715+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2229}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4008","reported_seq":4080,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.333905+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:54.333905+0000","last_peered":"2024-09-10T20:21:54.333905+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:54.333905+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32789664,"num_objects":4003,"num_object_clones":0,"num_object_copies":16012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4006,"num_write_kb":32036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530373936,"num_objects":64747,"num_object_clones":0,"num_object_copies":258988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57778,"num_write_kb":462133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":948305920,"data_stored":948305920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64806,"ondisk_log_size":64806,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478540,"kb_used_data":258044,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93893300,"statfs":{"total":96636764160,"available":96146739200,"internally_reserved":0,"allocated":264237056,"data_stored":264042683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481948,"kb_used_data":261964,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93889892,"statfs":{"total":96636764160,"available":96143249408,"internally_reserved":0,"allocated":268251136,"data_stored":268056763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309620,"kb_used_data":147428,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94062220,"statfs":{"total":96636764160,"available":96319713280,"internally_reserved":0,"allocated":150966272,"data_stored":150771899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473376,"kb_used_data":257104,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93898464,"statfs":{"total":96636764160,"available":96152027136,"internally_reserved":0,"allocated":263274496,"data_stored":263080123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264101888,"data_stored":264101888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151420928,"data_stored":151420928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267948032,"data_stored":267948032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":264835072,"data_stored":264835072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:21:59.936 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:21:59.936 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4043", 'reported_seq': 4093, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.835246+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:54.835246+0000', 'last_peered': '2024-09-10T20:21:54.835246+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:54.835246+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4043, 'log_dups_size': 0, 'ondisk_log_size': 4043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33120256, 'num_objects': 4043, 'num_object_clones': 0, 'num_object_copies': 16172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4043, 'num_write_kb': 32344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2077}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:59.936 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:21:59.936 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4099", 'reported_seq': 4183, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:57.046918+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:21:57.046918+0000', 'last_peered': '2024-09-10T20:21:57.046918+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:21:57.046918+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4099, 'log_dups_size': 0, 'ondisk_log_size': 4099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33552976, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 16384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4098, 'num_write_kb': 32774, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2134}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4126", 'reported_seq': 5194, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.388539+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:21:54.388539+0000', 'last_peered': '2024-09-10T20:21:54.388539+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:54.388539+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4126, 'log_dups_size': 0, 'ondisk_log_size': 4126, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33764512, 'num_objects': 4122, 'num_object_clones': 0, 'num_object_copies': 16488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1505, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4124, 'num_write_kb': 32981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 500, 'num_bytes_recovered': 4079616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4064", 'reported_seq': 4123, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.756741+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:54.756741+0000', 'last_peered': '2024-09-10T20:21:54.756741+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:54.756741+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4064, 'log_dups_size': 0, 'ondisk_log_size': 4064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33264800, 'num_objects': 4061, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4062, 'num_write_kb': 32493, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2081}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4073", 'reported_seq': 4123, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.611753+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:54.611753+0000', 'last_peered': '2024-09-10T20:21:54.611753+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:54.611753+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4073, 'log_dups_size': 0, 'ondisk_log_size': 4073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33366016, 'num_objects': 4073, 'num_object_clones': 0, 'num_object_copies': 16292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4073, 'num_write_kb': 32584, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2135}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:21:59.937 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4056", 'reported_seq': 4129, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.259693+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:54.259693+0000', 'last_peered': '2024-09-10T20:21:54.259693+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:54.259693+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4056, 'log_dups_size': 0, 'ondisk_log_size': 4056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33174688, 'num_objects': 4050, 'num_object_clones': 0, 'num_object_copies': 16200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4054, 'num_write_kb': 32412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2097}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4078", 'reported_seq': 4150, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.161583+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:54.161583+0000', 'last_peered': '2024-09-10T20:21:54.161583+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:54.161583+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4078, 'log_dups_size': 0, 'ondisk_log_size': 4078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33363104, 'num_objects': 4073, 'num_object_clones': 0, 'num_object_copies': 16292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4076, 'num_write_kb': 32596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2030}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4016", 'reported_seq': 4101, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:57.098959+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:21:57.098959+0000', 'last_peered': '2024-09-10T20:21:57.098959+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:21:57.098959+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4016, 'log_dups_size': 0, 'ondisk_log_size': 4016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32879776, 'num_objects': 4014, 'num_object_clones': 0, 'num_object_copies': 16056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4014, 'num_write_kb': 32110, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2111}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'3992", 'reported_seq': 4055, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.667957+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:54.667957+0000', 'last_peered': '2024-09-10T20:21:54.667957+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:54.667957+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3992, 'log_dups_size': 0, 'ondisk_log_size': 3992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32657136, 'num_objects': 3987, 'num_object_clones': 0, 'num_object_copies': 15948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3989, 'num_write_kb': 31900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4058", 'reported_seq': 4107, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.849480+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:54.849480+0000', 'last_peered': '2024-09-10T20:21:54.849480+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:54.849480+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4058, 'log_dups_size': 0, 'ondisk_log_size': 4058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33233488, 'num_objects': 4057, 'num_object_clones': 0, 'num_object_copies': 16228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4057, 'num_write_kb': 32455, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:21:59.938 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'3943", 'reported_seq': 4023, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:56.828502+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:21:56.828502+0000', 'last_peered': '2024-09-10T20:21:56.828502+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:21:56.828502+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 3943, 'log_dups_size': 0, 'ondisk_log_size': 3943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32291408, 'num_objects': 3942, 'num_object_clones': 0, 'num_object_copies': 15768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3942, 'num_write_kb': 31535, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1994}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:59.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:21:59.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4047", 'reported_seq': 4104, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.880091+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:54.880091+0000', 'last_peered': '2024-09-10T20:21:54.880091+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:54.880091+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4047, 'log_dups_size': 0, 'ondisk_log_size': 4047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33117344, 'num_objects': 4043, 'num_object_clones': 0, 'num_object_copies': 16172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4045, 'num_write_kb': 32356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2068}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:21:59.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:21:59.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4194", 'reported_seq': 4281, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:56.898715+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:21:56.898715+0000', 'last_peered': '2024-09-10T20:21:56.898715+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:21:56.898715+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4194, 'log_dups_size': 0, 'ondisk_log_size': 4194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34329760, 'num_objects': 4191, 'num_object_clones': 0, 'num_object_copies': 16764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4192, 'num_write_kb': 33533, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2229}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:21:59.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:21:59.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4008", 'reported_seq': 4080, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:54.333905+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:54.333905+0000', 'last_peered': '2024-09-10T20:21:54.333905+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:54.333905+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4008, 'log_dups_size': 0, 'ondisk_log_size': 4008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32789664, 'num_objects': 4003, 'num_object_clones': 0, 'num_object_copies': 16012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4006, 'num_write_kb': 32036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:21:59.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:00.648 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:00.648 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26970/261156 objects degraded (10.327%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":209,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65289,"data_bytes":534814000,"bytes_used":1808883712,"bytes_avail":384738172928,"bytes_total":386547056640,"degraded_objects":26970,"degraded_total":261156,"degraded_ratio":0.10327160777466342,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":191097,"recovering_keys_per_sec":0,"num_objects_recovered":280,"num_bytes_recovered":2293760,"num_keys_recovered":0,"write_bytes_sec":2669907,"read_op_per_sec":0,"write_op_per_sec":325},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:00.670 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:00.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:00.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 139 20 66636 66616 3.74366 2.53906 0.0837537 0.0417214 2024-09-10T20:22:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:01.249 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:01.271 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-10T20:21:59.922311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534814000,"num_objects":65289,"num_object_clones":0,"num_object_copies":261156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58320,"num_write_kb":466469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65348,"ondisk_log_size":65348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1766488,"kb_used_data":939224,"kb_used_omap":14,"kb_used_meta":827185,"kb_avail":375720872,"statfs":{"total":386547056640,"available":384738172928,"internally_reserved":0,"allocated":961765376,"data_stored":960987884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":847038006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,4,16,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32047104,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3912,"num_write_kb":31296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003078"},"pg_stats":[{"pgid":"1.f","version":"36'4043","reported_seq":4093,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.835246+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:54.835246+0000","last_peered":"2024-09-10T20:21:54.835246+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:54.835246+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33120256,"num_objects":4043,"num_object_clones":0,"num_object_copies":16172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4135","reported_seq":4257,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:57.695597+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:21:57.695597+0000","last_peered":"2024-09-10T20:21:57.695597+0000","last_clean":"2024-09-10T20:21:57.695597+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:21:57.695597+0000","last_undegraded":"2024-09-10T20:21:57.695597+0000","last_fullsized":"2024-09-10T20:21:57.695597+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33824768,"num_objects":4129,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":576,"num_write_kb":4608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4099","reported_seq":4183,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:57.046918+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:21:57.046918+0000","last_peered":"2024-09-10T20:21:57.046918+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:21:57.046918+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33552976,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4222","reported_seq":5485,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.405314+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:59.405314+0000","last_peered":"2024-09-10T20:21:59.405314+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:59.405314+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34550944,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":33749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4064","reported_seq":4123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.756741+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:54.756741+0000","last_peered":"2024-09-10T20:21:54.756741+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:54.756741+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264800,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":32493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4073","reported_seq":4123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.611753+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:54.611753+0000","last_peered":"2024-09-10T20:21:54.611753+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:54.611753+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33366016,"num_objects":4073,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":32584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4147","reported_seq":4220,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.237324+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:59.237324+0000","last_peered":"2024-09-10T20:21:59.237324+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:59.237324+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33920160,"num_objects":4141,"num_object_clones":0,"num_object_copies":16564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4145,"num_write_kb":33140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2188}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4195","reported_seq":4266,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.332868+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:59.332868+0000","last_peered":"2024-09-10T20:21:59.332868+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:59.332868+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34321568,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3992","reported_seq":7954,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:54.893708+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:54.893708+0000","last_peered":"2024-09-10T20:21:54.893708+0000","last_clean":"2024-09-10T20:21:54.893708+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:54.893708+0000","last_undegraded":"2024-09-10T20:21:54.893708+0000","last_fullsized":"2024-09-10T20:21:54.893708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32610896,"num_objects":3981,"num_object_clones":0,"num_object_copies":15924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":545,"num_write_kb":4360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4016","reported_seq":4101,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:57.098959+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:21:57.098959+0000","last_peered":"2024-09-10T20:21:57.098959+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:21:57.098959+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32879776,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":32110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'3992","reported_seq":4055,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.667957+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:54.667957+0000","last_peered":"2024-09-10T20:21:54.667957+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:54.667957+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32657136,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":31900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4058","reported_seq":4107,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.849480+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:54.849480+0000","last_peered":"2024-09-10T20:21:54.849480+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:54.849480+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33233488,"num_objects":4057,"num_object_clones":0,"num_object_copies":16228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":32455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3943","reported_seq":4023,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:56.828502+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:21:56.828502+0000","last_peered":"2024-09-10T20:21:56.828502+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:21:56.828502+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32291408,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":31535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4047","reported_seq":4104,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:54.880091+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:54.880091+0000","last_peered":"2024-09-10T20:21:54.880091+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:54.880091+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33117344,"num_objects":4043,"num_object_clones":0,"num_object_copies":16172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4045,"num_write_kb":32356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2068}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4194","reported_seq":4281,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:56.898715+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:21:56.898715+0000","last_peered":"2024-09-10T20:21:56.898715+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:21:56.898715+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2229}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4128","reported_seq":4200,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.303189+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:59.303189+0000","last_peered":"2024-09-10T20:21:59.303189+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:59.303189+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772704,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4126,"num_write_kb":32996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2144}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534814000,"num_objects":65289,"num_object_clones":0,"num_object_copies":261156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58320,"num_write_kb":466469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":962584576,"data_stored":962584576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65348,"ondisk_log_size":65348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478540,"kb_used_data":258044,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93893300,"statfs":{"total":96636764160,"available":96146739200,"internally_reserved":0,"allocated":264237056,"data_stored":264042683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481948,"kb_used_data":261964,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93889892,"statfs":{"total":96636764160,"available":96143249408,"internally_reserved":0,"allocated":268251136,"data_stored":268056763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320652,"kb_used_data":154300,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94051188,"statfs":{"total":96636764160,"available":96308416512,"internally_reserved":0,"allocated":158003200,"data_stored":157808827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170323666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485348,"kb_used_data":264916,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93886492,"statfs":{"total":96636764160,"available":96139767808,"internally_reserved":0,"allocated":271273984,"data_stored":271079611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,9,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271122432,"data_stored":271122432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158679040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267948032,"data_stored":267948032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":264835072,"data_stored":264835072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:01.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:22:01.785488+0000 min lat: 0.00230373 max lat: 12.2408 avg lat: 0.0418004 2024-09-10T20:22:01.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:22:01.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 140 20 66990 66970 3.73667 2.76562 0.0798808 0.0418004 2024-09-10T20:22:02.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 141 20 67318 67298 3.72834 2.5625 0.0399585 0.0418935 2024-09-10T20:22:03.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 142 20 67635 67615 3.71952 2.47656 0.0829128 0.0419961 2024-09-10T20:22:04.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:04.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 143 20 67978 67958 3.71225 2.67969 0.0556578 0.0420763 2024-09-10T20:22:04.871 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:04.872 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:04.893 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-10T20:22:03.923471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545537328,"num_objects":66598,"num_object_clones":0,"num_object_copies":266392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59629,"num_write_kb":476941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66657,"ondisk_log_size":66657,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1766320,"kb_used_data":959856,"kb_used_omap":14,"kb_used_meta":806385,"kb_avail":375721040,"statfs":{"total":386547056640,"available":384738344960,"internally_reserved":0,"allocated":982892544,"data_stored":982115052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":825738806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,13,1,20,23,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32374784,"num_objects":3952,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003323"},"pg_stats":[{"pgid":"1.f","version":"36'4149","reported_seq":4199,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.876536+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:59.876536+0000","last_peered":"2024-09-10T20:21:59.876536+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:59.876536+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33988608,"num_objects":4149,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":33192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4243","reported_seq":4365,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:03.007526+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:03.007526+0000","last_peered":"2024-09-10T20:22:03.007526+0000","last_clean":"2024-09-10T20:22:03.007526+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:03.007526+0000","last_undegraded":"2024-09-10T20:22:03.007526+0000","last_fullsized":"2024-09-10T20:22:03.007526+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34709504,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":684,"num_write_kb":5472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4205","reported_seq":4289,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.248162+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:02.248162+0000","last_peered":"2024-09-10T20:22:02.248162+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:02.248162+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34421328,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2240}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4222","reported_seq":5485,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.405314+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:59.405314+0000","last_peered":"2024-09-10T20:21:59.405314+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:59.405314+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34550944,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":33749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4160","reported_seq":4220,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.824243+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:59.824243+0000","last_peered":"2024-09-10T20:21:59.824243+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:59.824243+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34051232,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4192","reported_seq":4242,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.874695+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:59.874695+0000","last_peered":"2024-09-10T20:21:59.874695+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:59.874695+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4147","reported_seq":4220,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.237324+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:59.237324+0000","last_peered":"2024-09-10T20:21:59.237324+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:59.237324+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33920160,"num_objects":4141,"num_object_clones":0,"num_object_copies":16564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4145,"num_write_kb":33140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2188}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4195","reported_seq":4266,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.332868+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:59.332868+0000","last_peered":"2024-09-10T20:21:59.332868+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:59.332868+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34321568,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4104","reported_seq":8066,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:59.899485+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:59.899485+0000","last_peered":"2024-09-10T20:21:59.899485+0000","last_clean":"2024-09-10T20:21:59.899485+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:59.899485+0000","last_undegraded":"2024-09-10T20:21:59.899485+0000","last_fullsized":"2024-09-10T20:21:59.899485+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4104,"log_dups_size":0,"ondisk_log_size":4104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528400,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":657,"num_write_kb":5256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4129","reported_seq":4214,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.158970+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:02.158970+0000","last_peered":"2024-09-10T20:22:02.158970+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:02.158970+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805472,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4120","reported_seq":4182,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.918676+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:59.918676+0000","last_peered":"2024-09-10T20:21:59.918676+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:59.918676+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33705712,"num_objects":4115,"num_object_clones":0,"num_object_copies":16460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4159","reported_seq":4211,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.748045+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:59.748045+0000","last_peered":"2024-09-10T20:21:59.748045+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:59.748045+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4028","reported_seq":4111,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.250637+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:02.250637+0000","last_peered":"2024-09-10T20:22:02.250637+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:02.250637+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32987728,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4027,"num_write_kb":32215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4162","reported_seq":4221,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.915718+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:59.915718+0000","last_peered":"2024-09-10T20:21:59.915718+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:59.915718+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4162,"log_dups_size":0,"ondisk_log_size":4162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059424,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":33276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4314","reported_seq":4401,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.230786+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:02.230786+0000","last_peered":"2024-09-10T20:22:02.230786+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:02.230786+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35312800,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4128","reported_seq":4200,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.303189+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:59.303189+0000","last_peered":"2024-09-10T20:21:59.303189+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:59.303189+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772704,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4126,"num_write_kb":32996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2144}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":545537328,"num_objects":66598,"num_object_clones":0,"num_object_copies":266392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59629,"num_write_kb":476941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983904256,"data_stored":983904256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66657,"ondisk_log_size":66657,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489476,"kb_used_data":264820,"kb_used_omap":4,"kb_used_meta":224635,"kb_avail":93882364,"statfs":{"total":96636764160,"available":96135540736,"internally_reserved":0,"allocated":271175680,"data_stored":270981307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":230026960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,9,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476192,"kb_used_data":268688,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93895648,"statfs":{"total":96636764160,"available":96149143552,"internally_reserved":0,"allocated":275136512,"data_stored":274942139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315304,"kb_used_data":161432,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":94056536,"statfs":{"total":96636764160,"available":96313892864,"internally_reserved":0,"allocated":165306368,"data_stored":165111995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":157544146},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,11,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485348,"kb_used_data":264916,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93886492,"statfs":{"total":96636764160,"available":96139767808,"internally_reserved":0,"allocated":271273984,"data_stored":271079611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,9,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271122432,"data_stored":271122432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165998592,"data_stored":165998592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274984960,"data_stored":274984960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271798272,"data_stored":271798272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:04.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:05.363 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:05.383 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-10T20:22:03.923471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545537328,"num_objects":66598,"num_object_clones":0,"num_object_copies":266392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59629,"num_write_kb":476941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66657,"ondisk_log_size":66657,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1766320,"kb_used_data":959856,"kb_used_omap":14,"kb_used_meta":806385,"kb_avail":375721040,"statfs":{"total":386547056640,"available":384738344960,"internally_reserved":0,"allocated":982892544,"data_stored":982115052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":825738806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,13,1,20,23,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32374784,"num_objects":3952,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003323"},"pg_stats":[{"pgid":"1.f","version":"36'4149","reported_seq":4199,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.876536+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:21:59.876536+0000","last_peered":"2024-09-10T20:21:59.876536+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:21:59.876536+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33988608,"num_objects":4149,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":33192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4243","reported_seq":4365,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:03.007526+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:03.007526+0000","last_peered":"2024-09-10T20:22:03.007526+0000","last_clean":"2024-09-10T20:22:03.007526+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:03.007526+0000","last_undegraded":"2024-09-10T20:22:03.007526+0000","last_fullsized":"2024-09-10T20:22:03.007526+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34709504,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":684,"num_write_kb":5472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4205","reported_seq":4289,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.248162+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:02.248162+0000","last_peered":"2024-09-10T20:22:02.248162+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:02.248162+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34421328,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2240}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4222","reported_seq":5485,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.405314+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:21:59.405314+0000","last_peered":"2024-09-10T20:21:59.405314+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:21:59.405314+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34550944,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":33749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4160","reported_seq":4220,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.824243+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:21:59.824243+0000","last_peered":"2024-09-10T20:21:59.824243+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:21:59.824243+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34051232,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4192","reported_seq":4242,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.874695+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:21:59.874695+0000","last_peered":"2024-09-10T20:21:59.874695+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:21:59.874695+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4147","reported_seq":4220,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.237324+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:21:59.237324+0000","last_peered":"2024-09-10T20:21:59.237324+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:21:59.237324+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33920160,"num_objects":4141,"num_object_clones":0,"num_object_copies":16564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4145,"num_write_kb":33140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2188}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4195","reported_seq":4266,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.332868+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:21:59.332868+0000","last_peered":"2024-09-10T20:21:59.332868+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:21:59.332868+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34321568,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4104","reported_seq":8066,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:21:59.899485+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:21:59.899485+0000","last_peered":"2024-09-10T20:21:59.899485+0000","last_clean":"2024-09-10T20:21:59.899485+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:21:59.899485+0000","last_undegraded":"2024-09-10T20:21:59.899485+0000","last_fullsized":"2024-09-10T20:21:59.899485+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4104,"log_dups_size":0,"ondisk_log_size":4104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528400,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":657,"num_write_kb":5256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4129","reported_seq":4214,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.158970+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:02.158970+0000","last_peered":"2024-09-10T20:22:02.158970+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:02.158970+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805472,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4120","reported_seq":4182,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.918676+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:21:59.918676+0000","last_peered":"2024-09-10T20:21:59.918676+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:21:59.918676+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33705712,"num_objects":4115,"num_object_clones":0,"num_object_copies":16460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4159","reported_seq":4211,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.748045+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:21:59.748045+0000","last_peered":"2024-09-10T20:21:59.748045+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:21:59.748045+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4028","reported_seq":4111,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.250637+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:02.250637+0000","last_peered":"2024-09-10T20:22:02.250637+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:02.250637+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32987728,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4027,"num_write_kb":32215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4162","reported_seq":4221,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.915718+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:21:59.915718+0000","last_peered":"2024-09-10T20:21:59.915718+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:21:59.915718+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4162,"log_dups_size":0,"ondisk_log_size":4162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059424,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":33276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4314","reported_seq":4401,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.230786+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:02.230786+0000","last_peered":"2024-09-10T20:22:02.230786+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:02.230786+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35312800,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4128","reported_seq":4200,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:21:59.303189+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:21:59.303189+0000","last_peered":"2024-09-10T20:21:59.303189+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:21:59.303189+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772704,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4126,"num_write_kb":32996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2144}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":545537328,"num_objects":66598,"num_object_clones":0,"num_object_copies":266392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59629,"num_write_kb":476941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":4882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983904256,"data_stored":983904256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66657,"ondisk_log_size":66657,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489476,"kb_used_data":264820,"kb_used_omap":4,"kb_used_meta":224635,"kb_avail":93882364,"statfs":{"total":96636764160,"available":96135540736,"internally_reserved":0,"allocated":271175680,"data_stored":270981307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":230026960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,9,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476192,"kb_used_data":268688,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93895648,"statfs":{"total":96636764160,"available":96149143552,"internally_reserved":0,"allocated":275136512,"data_stored":274942139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315304,"kb_used_data":161432,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":94056536,"statfs":{"total":96636764160,"available":96313892864,"internally_reserved":0,"allocated":165306368,"data_stored":165111995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":157544146},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,11,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485348,"kb_used_data":264916,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93886492,"statfs":{"total":96636764160,"available":96139767808,"internally_reserved":0,"allocated":271273984,"data_stored":271079611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,9,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271122432,"data_stored":271122432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165998592,"data_stored":165998592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274984960,"data_stored":274984960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271798272,"data_stored":271798272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:05.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:05.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4149", 'reported_seq': 4199, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.876536+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:21:59.876536+0000', 'last_peered': '2024-09-10T20:21:59.876536+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:21:59.876536+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4149, 'log_dups_size': 0, 'ondisk_log_size': 4149, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33988608, 'num_objects': 4149, 'num_object_clones': 0, 'num_object_copies': 16596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4149, 'num_write_kb': 33192, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:05.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:05.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4205", 'reported_seq': 4289, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:02.248162+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:02.248162+0000', 'last_peered': '2024-09-10T20:22:02.248162+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:02.248162+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4205, 'log_dups_size': 0, 'ondisk_log_size': 4205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34421328, 'num_objects': 4202, 'num_object_clones': 0, 'num_object_copies': 16808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4204, 'num_write_kb': 33622, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2240}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:05.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:05.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4222", 'reported_seq': 5485, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.405314+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:21:59.405314+0000', 'last_peered': '2024-09-10T20:21:59.405314+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:21:59.405314+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4222, 'log_dups_size': 0, 'ondisk_log_size': 4222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34550944, 'num_objects': 4218, 'num_object_clones': 0, 'num_object_copies': 16872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1407, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4220, 'num_write_kb': 33749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 598, 'num_bytes_recovered': 4882432, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:05.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4160", 'reported_seq': 4220, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.824243+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:21:59.824243+0000', 'last_peered': '2024-09-10T20:21:59.824243+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:21:59.824243+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4160, 'log_dups_size': 0, 'ondisk_log_size': 4160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34051232, 'num_objects': 4157, 'num_object_clones': 0, 'num_object_copies': 16628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4158, 'num_write_kb': 33261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4192", 'reported_seq': 4242, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.874695+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:21:59.874695+0000', 'last_peered': '2024-09-10T20:21:59.874695+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:21:59.874695+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4192, 'log_dups_size': 0, 'ondisk_log_size': 4192, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34340864, 'num_objects': 4192, 'num_object_clones': 0, 'num_object_copies': 16768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4192, 'num_write_kb': 33536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2254}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4147", 'reported_seq': 4220, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.237324+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:21:59.237324+0000', 'last_peered': '2024-09-10T20:21:59.237324+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:21:59.237324+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4147, 'log_dups_size': 0, 'ondisk_log_size': 4147, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33920160, 'num_objects': 4141, 'num_object_clones': 0, 'num_object_copies': 16564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4145, 'num_write_kb': 33140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2188}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4195", 'reported_seq': 4266, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.332868+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:21:59.332868+0000', 'last_peered': '2024-09-10T20:21:59.332868+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:21:59.332868+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4195, 'log_dups_size': 0, 'ondisk_log_size': 4195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34321568, 'num_objects': 4190, 'num_object_clones': 0, 'num_object_copies': 16760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4193, 'num_write_kb': 33532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4129", 'reported_seq': 4214, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:02.158970+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:02.158970+0000', 'last_peered': '2024-09-10T20:22:02.158970+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:02.158970+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4129, 'log_dups_size': 0, 'ondisk_log_size': 4129, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33805472, 'num_objects': 4127, 'num_object_clones': 0, 'num_object_copies': 16508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4127, 'num_write_kb': 33014, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2224}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:05.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4120", 'reported_seq': 4182, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.918676+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:21:59.918676+0000', 'last_peered': '2024-09-10T20:21:59.918676+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:21:59.918676+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4120, 'log_dups_size': 0, 'ondisk_log_size': 4120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33705712, 'num_objects': 4115, 'num_object_clones': 0, 'num_object_copies': 16460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4117, 'num_write_kb': 32924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2159}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4159", 'reported_seq': 4211, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.748045+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:21:59.748045+0000', 'last_peered': '2024-09-10T20:21:59.748045+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:21:59.748045+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4159, 'log_dups_size': 0, 'ondisk_log_size': 4159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34060880, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4158, 'num_write_kb': 33263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4028", 'reported_seq': 4111, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:02.250637+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:02.250637+0000', 'last_peered': '2024-09-10T20:22:02.250637+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:02.250637+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4028, 'log_dups_size': 0, 'ondisk_log_size': 4028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32987728, 'num_objects': 4027, 'num_object_clones': 0, 'num_object_copies': 16108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4027, 'num_write_kb': 32215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4162", 'reported_seq': 4221, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.915718+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:21:59.915718+0000', 'last_peered': '2024-09-10T20:21:59.915718+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:21:59.915718+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4162, 'log_dups_size': 0, 'ondisk_log_size': 4162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34059424, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4160, 'num_write_kb': 33276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4314", 'reported_seq': 4401, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:02.230786+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:02.230786+0000', 'last_peered': '2024-09-10T20:22:02.230786+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:02.230786+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4314, 'log_dups_size': 0, 'ondisk_log_size': 4314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35312800, 'num_objects': 4311, 'num_object_clones': 0, 'num_object_copies': 17244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4312, 'num_write_kb': 34493, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2349}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:05.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4128", 'reported_seq': 4200, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:21:59.303189+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:21:59.303189+0000', 'last_peered': '2024-09-10T20:21:59.303189+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:21:59.303189+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4128, 'log_dups_size': 0, 'ondisk_log_size': 4128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33772704, 'num_objects': 4123, 'num_object_clones': 0, 'num_object_copies': 16492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4126, 'num_write_kb': 32996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2144}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:05.387 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:05.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 144 20 68261 68241 3.70182 2.21094 0.0386116 0.0421943 2024-09-10T20:22:06.120 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:06.121 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26970/266392 objects degraded (10.124%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":214,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66598,"data_bytes":545537328,"bytes_used":1808711680,"bytes_avail":384738344960,"bytes_total":386547056640,"degraded_objects":26970,"degraded_total":266392,"degraded_ratio":0.10124177903240338,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":149462,"recovering_keys_per_sec":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"write_bytes_sec":2697151,"read_op_per_sec":0,"write_op_per_sec":329},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:06.141 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:06.141 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:06.725 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:06.725 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:06.745 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-10T20:22:05.924108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":554597680,"num_objects":67704,"num_object_clones":0,"num_object_copies":270816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60735,"num_write_kb":485789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67763,"ondisk_log_size":67763,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1754640,"kb_used_data":973136,"kb_used_omap":14,"kb_used_meta":781425,"kb_avail":375732720,"statfs":{"total":386547056640,"available":384750305280,"internally_reserved":0,"allocated":996491264,"data_stored":995713772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":800179766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,12,7,13,22,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37101568,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4529,"num_write_kb":36232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003476"},"pg_stats":[{"pgid":"1.f","version":"36'4247","reported_seq":4297,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.758867+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:04.758867+0000","last_peered":"2024-09-10T20:22:04.758867+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:04.758867+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34791424,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":33976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4243","reported_seq":4365,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:03.007526+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:03.007526+0000","last_peered":"2024-09-10T20:22:03.007526+0000","last_clean":"2024-09-10T20:22:03.007526+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:03.007526+0000","last_undegraded":"2024-09-10T20:22:03.007526+0000","last_fullsized":"2024-09-10T20:22:03.007526+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34709504,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":684,"num_write_kb":5472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4205","reported_seq":4289,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.248162+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:02.248162+0000","last_peered":"2024-09-10T20:22:02.248162+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:02.248162+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34421328,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2240}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4312","reported_seq":5704,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.385447+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:04.385447+0000","last_peered":"2024-09-10T20:22:04.385447+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:04.385447+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288224,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":34469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4270","reported_seq":4329,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.831332+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:04.831332+0000","last_peered":"2024-09-10T20:22:04.831332+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:04.831332+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34952352,"num_objects":4267,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4268,"num_write_kb":34141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4288","reported_seq":4338,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.819671+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:04.819671+0000","last_peered":"2024-09-10T20:22:04.819671+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:04.819671+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4243","reported_seq":4316,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.388861+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:04.388861+0000","last_peered":"2024-09-10T20:22:04.388861+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:04.388861+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34706592,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2284}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4295","reported_seq":4367,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.272289+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:04.272289+0000","last_peered":"2024-09-10T20:22:04.272289+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:04.272289+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35140768,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4293,"num_write_kb":34332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4219","reported_seq":8181,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:04.747959+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:04.747959+0000","last_peered":"2024-09-10T20:22:04.747959+0000","last_clean":"2024-09-10T20:22:04.747959+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:04.747959+0000","last_undegraded":"2024-09-10T20:22:04.747959+0000","last_fullsized":"2024-09-10T20:22:04.747959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34470480,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":772,"num_write_kb":6176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4129","reported_seq":4214,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.158970+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:02.158970+0000","last_peered":"2024-09-10T20:22:02.158970+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:02.158970+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805472,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4224","reported_seq":4287,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.815845+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:04.815845+0000","last_peered":"2024-09-10T20:22:04.815845+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:04.815845+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34557680,"num_objects":4219,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":33756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4252","reported_seq":4304,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.748683+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:04.748683+0000","last_peered":"2024-09-10T20:22:04.748683+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:04.748683+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34822736,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":34007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4028","reported_seq":4111,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.250637+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:02.250637+0000","last_peered":"2024-09-10T20:22:02.250637+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:02.250637+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32987728,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4027,"num_write_kb":32215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4264","reported_seq":4323,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.443990+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:04.443990+0000","last_peered":"2024-09-10T20:22:04.443990+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:04.443990+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34895008,"num_objects":4260,"num_object_clones":0,"num_object_copies":17040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4262,"num_write_kb":34092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2285}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4314","reported_seq":4401,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:02.230786+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:02.230786+0000","last_peered":"2024-09-10T20:22:02.230786+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:02.230786+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35312800,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4230","reported_seq":4302,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.329488+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:04.329488+0000","last_peered":"2024-09-10T20:22:04.329488+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:04.329488+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34608288,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":33812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":554597680,"num_objects":67704,"num_object_clones":0,"num_object_copies":270816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60735,"num_write_kb":485789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":997494784,"data_stored":997494784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67763,"ondisk_log_size":67763,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483680,"kb_used_data":271504,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93888160,"statfs":{"total":96636764160,"available":96141475840,"internally_reserved":0,"allocated":278020096,"data_stored":277825723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476192,"kb_used_data":268688,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93895648,"statfs":{"total":96636764160,"available":96149143552,"internally_reserved":0,"allocated":275136512,"data_stored":274942139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315304,"kb_used_data":161432,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":94056536,"statfs":{"total":96636764160,"available":96313892864,"internally_reserved":0,"allocated":165306368,"data_stored":165111995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":157544146},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,11,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479464,"kb_used_data":271512,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93892376,"statfs":{"total":96636764160,"available":96145793024,"internally_reserved":0,"allocated":278028288,"data_stored":277833915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212922063},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277938176,"data_stored":277938176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165998592,"data_stored":165998592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274984960,"data_stored":274984960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278573056,"data_stored":278573056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:06.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 145 20 68584 68564 3.6937 2.52344 0.103105 0.0422907 2024-09-10T20:22:07.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 146 20 68954 68934 3.68819 2.89062 0.0498254 0.042351 2024-09-10T20:22:08.788 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 147 20 69317 69297 3.6824 2.83594 0.0464361 0.0424192 2024-09-10T20:22:09.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:09.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 148 20 69640 69620 3.67456 2.52344 0.0569087 0.042508 2024-09-10T20:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:10.377 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:10.406 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-10T20:22:09.925530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562437424,"num_objects":68661,"num_object_clones":0,"num_object_copies":274644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61692,"num_write_kb":493445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68720,"ondisk_log_size":68720,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1787164,"kb_used_data":993180,"kb_used_omap":14,"kb_used_meta":793905,"kb_avail":375700196,"statfs":{"total":386547056640,"available":384717000704,"internally_reserved":0,"allocated":1017016320,"data_stored":1016238828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":812959286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,5,17,12,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32063488,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":31312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":1990656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003765"},"pg_stats":[{"pgid":"1.f","version":"36'4247","reported_seq":4297,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.758867+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:04.758867+0000","last_peered":"2024-09-10T20:22:04.758867+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:04.758867+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34791424,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":33976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4351","reported_seq":4473,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:07.973150+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:07.973150+0000","last_peered":"2024-09-10T20:22:07.973150+0000","last_clean":"2024-09-10T20:22:07.973150+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:07.973150+0000","last_undegraded":"2024-09-10T20:22:07.973150+0000","last_fullsized":"2024-09-10T20:22:07.973150+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35594240,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":6336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4304","reported_seq":4388,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.295174+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:07.295174+0000","last_peered":"2024-09-10T20:22:07.295174+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:07.295174+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35232336,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4303,"num_write_kb":34414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4415","reported_seq":5969,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.405439+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:09.405439+0000","last_peered":"2024-09-10T20:22:09.405439+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:09.405439+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36132000,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4270","reported_seq":4329,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.831332+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:04.831332+0000","last_peered":"2024-09-10T20:22:04.831332+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:04.831332+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34952352,"num_objects":4267,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4268,"num_write_kb":34141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4288","reported_seq":4338,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.819671+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:04.819671+0000","last_peered":"2024-09-10T20:22:04.819671+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:04.819671+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4350","reported_seq":4422,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.443374+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:09.443374+0000","last_peered":"2024-09-10T20:22:09.443374+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:09.443374+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35583136,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4394","reported_seq":4466,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.345140+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:09.345140+0000","last_peered":"2024-09-10T20:22:09.345140+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:09.345140+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35951776,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":35124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4219","reported_seq":8181,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:04.747959+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:04.747959+0000","last_peered":"2024-09-10T20:22:04.747959+0000","last_clean":"2024-09-10T20:22:04.747959+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:04.747959+0000","last_undegraded":"2024-09-10T20:22:04.747959+0000","last_fullsized":"2024-09-10T20:22:04.747959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34470480,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":772,"num_write_kb":6176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4236","reported_seq":4321,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.177895+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:07.177895+0000","last_peered":"2024-09-10T20:22:07.177895+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:07.177895+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34682016,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4234,"num_write_kb":33870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4224","reported_seq":4287,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.815845+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:04.815845+0000","last_peered":"2024-09-10T20:22:04.815845+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:04.815845+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34557680,"num_objects":4219,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":33756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4252","reported_seq":4304,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.748683+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:04.748683+0000","last_peered":"2024-09-10T20:22:04.748683+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:04.748683+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34822736,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":34007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4157","reported_seq":4239,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.275834+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:07.275834+0000","last_peered":"2024-09-10T20:22:07.275834+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:07.275834+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34044496,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4264","reported_seq":4323,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.443990+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:04.443990+0000","last_peered":"2024-09-10T20:22:04.443990+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:04.443990+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34895008,"num_objects":4260,"num_object_clones":0,"num_object_copies":17040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4262,"num_write_kb":34092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2285}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4420","reported_seq":4507,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.185602+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:07.185602+0000","last_peered":"2024-09-10T20:22:07.185602+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:07.185602+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36181152,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4329","reported_seq":4401,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.447610+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:09.447610+0000","last_peered":"2024-09-10T20:22:09.447610+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:09.447610+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419296,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":34604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":562437424,"num_objects":68661,"num_object_clones":0,"num_object_copies":274644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61692,"num_write_kb":493445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1018097664,"data_stored":1018097664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68720,"ondisk_log_size":68720,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483680,"kb_used_data":271504,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93888160,"statfs":{"total":96636764160,"available":96141475840,"internally_reserved":0,"allocated":278020096,"data_stored":277825723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486964,"kb_used_data":275300,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93884876,"statfs":{"total":96636764160,"available":96138113024,"internally_reserved":0,"allocated":281907200,"data_stored":281712827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326300,"kb_used_data":168268,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94045540,"statfs":{"total":96636764160,"available":96302632960,"internally_reserved":0,"allocated":172306432,"data_stored":172112059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490220,"kb_used_data":278108,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881620,"statfs":{"total":96636764160,"available":96134778880,"internally_reserved":0,"allocated":284782592,"data_stored":284588219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284741632,"data_stored":284741632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173060096,"data_stored":173060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281722880,"data_stored":281722880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278573056,"data_stored":278573056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:10.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:10.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 149 20 69971 69951 3.66726 2.58594 0.0682663 0.0425929 2024-09-10T20:22:11.036 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:11.037 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:11.049 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-10T20:22:09.925530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562437424,"num_objects":68661,"num_object_clones":0,"num_object_copies":274644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61692,"num_write_kb":493445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68720,"ondisk_log_size":68720,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1787164,"kb_used_data":993180,"kb_used_omap":14,"kb_used_meta":793905,"kb_avail":375700196,"statfs":{"total":386547056640,"available":384717000704,"internally_reserved":0,"allocated":1017016320,"data_stored":1016238828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":812959286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,5,17,12,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32063488,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":31312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":1990656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003765"},"pg_stats":[{"pgid":"1.f","version":"36'4247","reported_seq":4297,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.758867+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:04.758867+0000","last_peered":"2024-09-10T20:22:04.758867+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:04.758867+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34791424,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":33976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4351","reported_seq":4473,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:07.973150+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:07.973150+0000","last_peered":"2024-09-10T20:22:07.973150+0000","last_clean":"2024-09-10T20:22:07.973150+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:07.973150+0000","last_undegraded":"2024-09-10T20:22:07.973150+0000","last_fullsized":"2024-09-10T20:22:07.973150+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35594240,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":6336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4304","reported_seq":4388,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.295174+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:07.295174+0000","last_peered":"2024-09-10T20:22:07.295174+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:07.295174+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35232336,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4303,"num_write_kb":34414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4415","reported_seq":5969,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.405439+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:09.405439+0000","last_peered":"2024-09-10T20:22:09.405439+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:09.405439+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36132000,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4270","reported_seq":4329,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.831332+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:04.831332+0000","last_peered":"2024-09-10T20:22:04.831332+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:04.831332+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34952352,"num_objects":4267,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4268,"num_write_kb":34141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4288","reported_seq":4338,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.819671+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:04.819671+0000","last_peered":"2024-09-10T20:22:04.819671+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:04.819671+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4350","reported_seq":4422,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.443374+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:09.443374+0000","last_peered":"2024-09-10T20:22:09.443374+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:09.443374+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35583136,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4394","reported_seq":4466,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.345140+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:09.345140+0000","last_peered":"2024-09-10T20:22:09.345140+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:09.345140+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35951776,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":35124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4219","reported_seq":8181,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:04.747959+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:04.747959+0000","last_peered":"2024-09-10T20:22:04.747959+0000","last_clean":"2024-09-10T20:22:04.747959+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:04.747959+0000","last_undegraded":"2024-09-10T20:22:04.747959+0000","last_fullsized":"2024-09-10T20:22:04.747959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34470480,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":772,"num_write_kb":6176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4236","reported_seq":4321,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.177895+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:07.177895+0000","last_peered":"2024-09-10T20:22:07.177895+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:07.177895+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34682016,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4234,"num_write_kb":33870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4224","reported_seq":4287,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.815845+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:04.815845+0000","last_peered":"2024-09-10T20:22:04.815845+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:04.815845+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34557680,"num_objects":4219,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":33756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4252","reported_seq":4304,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.748683+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:04.748683+0000","last_peered":"2024-09-10T20:22:04.748683+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:04.748683+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34822736,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":34007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4157","reported_seq":4239,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.275834+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:07.275834+0000","last_peered":"2024-09-10T20:22:07.275834+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:07.275834+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34044496,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4264","reported_seq":4323,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:04.443990+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:04.443990+0000","last_peered":"2024-09-10T20:22:04.443990+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:04.443990+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34895008,"num_objects":4260,"num_object_clones":0,"num_object_copies":17040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4262,"num_write_kb":34092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2285}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4420","reported_seq":4507,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.185602+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:07.185602+0000","last_peered":"2024-09-10T20:22:07.185602+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:07.185602+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36181152,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4329","reported_seq":4401,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.447610+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:09.447610+0000","last_peered":"2024-09-10T20:22:09.447610+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:09.447610+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419296,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":34604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":562437424,"num_objects":68661,"num_object_clones":0,"num_object_copies":274644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61692,"num_write_kb":493445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1018097664,"data_stored":1018097664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68720,"ondisk_log_size":68720,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483680,"kb_used_data":271504,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93888160,"statfs":{"total":96636764160,"available":96141475840,"internally_reserved":0,"allocated":278020096,"data_stored":277825723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486964,"kb_used_data":275300,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93884876,"statfs":{"total":96636764160,"available":96138113024,"internally_reserved":0,"allocated":281907200,"data_stored":281712827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326300,"kb_used_data":168268,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94045540,"statfs":{"total":96636764160,"available":96302632960,"internally_reserved":0,"allocated":172306432,"data_stored":172112059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490220,"kb_used_data":278108,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881620,"statfs":{"total":96636764160,"available":96134778880,"internally_reserved":0,"allocated":284782592,"data_stored":284588219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284741632,"data_stored":284741632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173060096,"data_stored":173060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281722880,"data_stored":281722880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278573056,"data_stored":278573056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:11.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:11.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4247", 'reported_seq': 4297, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:04.758867+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:04.758867+0000', 'last_peered': '2024-09-10T20:22:04.758867+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:04.758867+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4247, 'log_dups_size': 0, 'ondisk_log_size': 4247, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34791424, 'num_objects': 4247, 'num_object_clones': 0, 'num_object_copies': 16988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4247, 'num_write_kb': 33976, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2281}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:11.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:11.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4304", 'reported_seq': 4388, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:07.295174+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:07.295174+0000', 'last_peered': '2024-09-10T20:22:07.295174+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:07.295174+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4304, 'log_dups_size': 0, 'ondisk_log_size': 4304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35232336, 'num_objects': 4301, 'num_object_clones': 0, 'num_object_copies': 17204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4303, 'num_write_kb': 34414, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2339}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:11.051 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:11.051 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4415", 'reported_seq': 5969, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:09.405439+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:09.405439+0000', 'last_peered': '2024-09-10T20:22:09.405439+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:09.405439+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4415, 'log_dups_size': 0, 'ondisk_log_size': 4415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36132000, 'num_objects': 4411, 'num_object_clones': 0, 'num_object_copies': 17644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4411, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4413, 'num_write_kb': 35293, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 743, 'num_bytes_recovered': 6070272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:11.051 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:11.051 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4270", 'reported_seq': 4329, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:04.831332+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:04.831332+0000', 'last_peered': '2024-09-10T20:22:04.831332+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:04.831332+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4270, 'log_dups_size': 0, 'ondisk_log_size': 4270, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34952352, 'num_objects': 4267, 'num_object_clones': 0, 'num_object_copies': 17068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4268, 'num_write_kb': 34141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2287}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:11.051 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:11.051 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4288", 'reported_seq': 4338, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:04.819671+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:04.819671+0000', 'last_peered': '2024-09-10T20:22:04.819671+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:04.819671+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4288, 'log_dups_size': 0, 'ondisk_log_size': 4288, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35127296, 'num_objects': 4288, 'num_object_clones': 0, 'num_object_copies': 17152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4288, 'num_write_kb': 34304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2350}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:11.051 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4350", 'reported_seq': 4422, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:09.443374+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:09.443374+0000', 'last_peered': '2024-09-10T20:22:09.443374+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:09.443374+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4350, 'log_dups_size': 0, 'ondisk_log_size': 4350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35583136, 'num_objects': 4344, 'num_object_clones': 0, 'num_object_copies': 17376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4348, 'num_write_kb': 34764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2391}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4394", 'reported_seq': 4466, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:09.345140+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:09.345140+0000', 'last_peered': '2024-09-10T20:22:09.345140+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:09.345140+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4394, 'log_dups_size': 0, 'ondisk_log_size': 4394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35951776, 'num_objects': 4389, 'num_object_clones': 0, 'num_object_copies': 17556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4392, 'num_write_kb': 35124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4236", 'reported_seq': 4321, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:07.177895+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:07.177895+0000', 'last_peered': '2024-09-10T20:22:07.177895+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:07.177895+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4236, 'log_dups_size': 0, 'ondisk_log_size': 4236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34682016, 'num_objects': 4234, 'num_object_clones': 0, 'num_object_copies': 16936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4234, 'num_write_kb': 33870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2331}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4224", 'reported_seq': 4287, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:04.815845+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:04.815845+0000', 'last_peered': '2024-09-10T20:22:04.815845+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:04.815845+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4224, 'log_dups_size': 0, 'ondisk_log_size': 4224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34557680, 'num_objects': 4219, 'num_object_clones': 0, 'num_object_copies': 16876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4221, 'num_write_kb': 33756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4252", 'reported_seq': 4304, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:04.748683+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:04.748683+0000', 'last_peered': '2024-09-10T20:22:04.748683+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:04.748683+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4252, 'log_dups_size': 0, 'ondisk_log_size': 4252, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34822736, 'num_objects': 4251, 'num_object_clones': 0, 'num_object_copies': 17004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4251, 'num_write_kb': 34007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:11.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4157", 'reported_seq': 4239, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:07.275834+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:07.275834+0000', 'last_peered': '2024-09-10T20:22:07.275834+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:07.275834+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4157, 'log_dups_size': 0, 'ondisk_log_size': 4157, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34044496, 'num_objects': 4156, 'num_object_clones': 0, 'num_object_copies': 16624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4156, 'num_write_kb': 33247, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:11.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:11.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4264", 'reported_seq': 4323, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:04.443990+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:04.443990+0000', 'last_peered': '2024-09-10T20:22:04.443990+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:04.443990+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4264, 'log_dups_size': 0, 'ondisk_log_size': 4264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34895008, 'num_objects': 4260, 'num_object_clones': 0, 'num_object_copies': 17040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4262, 'num_write_kb': 34092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2285}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:11.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:11.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4420", 'reported_seq': 4507, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:07.185602+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:07.185602+0000', 'last_peered': '2024-09-10T20:22:07.185602+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:07.185602+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4420, 'log_dups_size': 0, 'ondisk_log_size': 4420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36181152, 'num_objects': 4417, 'num_object_clones': 0, 'num_object_copies': 17668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4418, 'num_write_kb': 35341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2455}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:11.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:11.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4329", 'reported_seq': 4401, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:09.447610+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:09.447610+0000', 'last_peered': '2024-09-10T20:22:09.447610+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:09.447610+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4329, 'log_dups_size': 0, 'ondisk_log_size': 4329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35419296, 'num_objects': 4324, 'num_object_clones': 0, 'num_object_copies': 17296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4327, 'num_write_kb': 34604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2345}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:11.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:11.728 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:11.728 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26825/274644 objects degraded (9.767%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68661,"data_bytes":562437424,"bytes_used":1830055936,"bytes_avail":384717000704,"bytes_total":386547056640,"degraded_objects":26825,"degraded_total":274644,"degraded_ratio":0.097671895253491792,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":165835,"recovering_keys_per_sec":0,"num_objects_recovered":243,"num_bytes_recovered":1990656,"num_keys_recovered":0,"write_bytes_sec":2671119,"read_op_per_sec":0,"write_op_per_sec":326},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:11.753 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:11.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:11.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 150 20 70251 70231 3.65739 2.1875 0.0715069 0.0427077 2024-09-10T20:22:12.264 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:12.264 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:12.291 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-10T20:22:11.926317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":568335664,"num_objects":69381,"num_object_clones":0,"num_object_copies":277524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62412,"num_write_kb":499205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69440,"ondisk_log_size":69440,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1798032,"kb_used_data":999888,"kb_used_omap":14,"kb_used_meta":798065,"kb_avail":375689328,"statfs":{"total":386547056640,"available":384705871872,"internally_reserved":0,"allocated":1023885312,"data_stored":1023107820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":817219126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,11,9,13,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33521664,"num_objects":4092,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4092,"num_write_kb":32736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004006"},"pg_stats":[{"pgid":"1.f","version":"36'4367","reported_seq":4416,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.803627+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:09.803627+0000","last_peered":"2024-09-10T20:22:09.803627+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:09.803627+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35774464,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4367,"num_write_kb":34936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4351","reported_seq":4473,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:07.973150+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:07.973150+0000","last_peered":"2024-09-10T20:22:07.973150+0000","last_clean":"2024-09-10T20:22:07.973150+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:07.973150+0000","last_undegraded":"2024-09-10T20:22:07.973150+0000","last_fullsized":"2024-09-10T20:22:07.973150+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35594240,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":6336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4304","reported_seq":4388,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.295174+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:07.295174+0000","last_peered":"2024-09-10T20:22:07.295174+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:07.295174+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35232336,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4303,"num_write_kb":34414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4415","reported_seq":5969,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.405439+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:09.405439+0000","last_peered":"2024-09-10T20:22:09.405439+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:09.405439+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36132000,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4367","reported_seq":4426,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.924406+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:09.924406+0000","last_peered":"2024-09-10T20:22:09.924406+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:09.924406+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35746976,"num_objects":4364,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":34917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4394","reported_seq":4444,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.933420+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:09.933420+0000","last_peered":"2024-09-10T20:22:09.933420+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:09.933420+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35995648,"num_objects":4394,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":35152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4350","reported_seq":4422,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.443374+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:09.443374+0000","last_peered":"2024-09-10T20:22:09.443374+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:09.443374+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35583136,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4394","reported_seq":4466,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.345140+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:09.345140+0000","last_peered":"2024-09-10T20:22:09.345140+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:09.345140+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35951776,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":35124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4326","reported_seq":8286,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:09.936061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:09.936061+0000","last_peered":"2024-09-10T20:22:09.936061+0000","last_clean":"2024-09-10T20:22:09.936061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:09.936061+0000","last_undegraded":"2024-09-10T20:22:09.936061+0000","last_fullsized":"2024-09-10T20:22:09.936061+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35347024,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":879,"num_write_kb":7032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4236","reported_seq":4321,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.177895+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:07.177895+0000","last_peered":"2024-09-10T20:22:07.177895+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:07.177895+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34682016,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4234,"num_write_kb":33870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4324","reported_seq":4387,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.897653+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:09.897653+0000","last_peered":"2024-09-10T20:22:09.897653+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:09.897653+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4324,"log_dups_size":0,"ondisk_log_size":4324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35376880,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4344","reported_seq":4396,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.879304+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:09.879304+0000","last_peered":"2024-09-10T20:22:09.879304+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:09.879304+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4344,"log_dups_size":0,"ondisk_log_size":4344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35576400,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2365}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4157","reported_seq":4239,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.275834+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:07.275834+0000","last_peered":"2024-09-10T20:22:07.275834+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:07.275834+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34044496,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4362","reported_seq":4421,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.805635+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:09.805635+0000","last_peered":"2024-09-10T20:22:09.805635+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:09.805635+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35697824,"num_objects":4358,"num_object_clones":0,"num_object_copies":17432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":34876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4420","reported_seq":4507,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:07.185602+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:07.185602+0000","last_peered":"2024-09-10T20:22:07.185602+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:07.185602+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36181152,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4329","reported_seq":4401,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.447610+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:09.447610+0000","last_peered":"2024-09-10T20:22:09.447610+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:09.447610+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419296,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":34604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":568335664,"num_objects":69381,"num_object_clones":0,"num_object_copies":277524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62412,"num_write_kb":499205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1025048576,"data_stored":1025048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69440,"ondisk_log_size":69440,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494548,"kb_used_data":278212,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93877292,"statfs":{"total":96636764160,"available":96130347008,"internally_reserved":0,"allocated":284889088,"data_stored":284694715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486964,"kb_used_data":275300,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93884876,"statfs":{"total":96636764160,"available":96138113024,"internally_reserved":0,"allocated":281907200,"data_stored":281712827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326300,"kb_used_data":168268,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94045540,"statfs":{"total":96636764160,"available":96302632960,"internally_reserved":0,"allocated":172306432,"data_stored":172112059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490220,"kb_used_data":278108,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881620,"statfs":{"total":96636764160,"available":96134778880,"internally_reserved":0,"allocated":284782592,"data_stored":284588219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284741632,"data_stored":284741632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173060096,"data_stored":173060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281722880,"data_stored":281722880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285523968,"data_stored":285523968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:12.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 151 20 70540 70520 3.64812 2.25781 0.0226625 0.0428156 2024-09-10T20:22:13.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 152 20 70914 70894 3.64334 2.92188 0.0544423 0.0428757 2024-09-10T20:22:14.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 153 20 71274 71254 3.63791 2.8125 0.0751706 0.0429383 2024-09-10T20:22:15.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:15.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 154 20 71532 71512 3.62737 2.01562 0.0570343 0.0430582 2024-09-10T20:22:15.805 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:15.834 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-10T20:22:13.926771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":572702000,"num_objects":69914,"num_object_clones":0,"num_object_copies":279656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62945,"num_write_kb":503469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69973,"ondisk_log_size":69973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1818520,"kb_used_data":1012056,"kb_used_omap":14,"kb_used_meta":806385,"kb_avail":375668840,"statfs":{"total":386547056640,"available":384684892160,"internally_reserved":0,"allocated":1036345344,"data_stored":1035567852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":825738806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,7,10,9,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31522816,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003926"},"pg_stats":[{"pgid":"1.f","version":"36'4367","reported_seq":4416,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.803627+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:09.803627+0000","last_peered":"2024-09-10T20:22:09.803627+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:09.803627+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35774464,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4367,"num_write_kb":34936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4462","reported_seq":4584,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:13.049544+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:13.049544+0000","last_peered":"2024-09-10T20:22:13.049544+0000","last_clean":"2024-09-10T20:22:13.049544+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:13.049544+0000","last_undegraded":"2024-09-10T20:22:13.049544+0000","last_fullsized":"2024-09-10T20:22:13.049544+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503552,"num_objects":4456,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":903,"num_write_kb":7224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4395","reported_seq":4479,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.292105+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:12.292105+0000","last_peered":"2024-09-10T20:22:12.292105+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:12.292105+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35977808,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":35142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4415","reported_seq":5969,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.405439+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:09.405439+0000","last_peered":"2024-09-10T20:22:09.405439+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:09.405439+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36132000,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4367","reported_seq":4426,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.924406+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:09.924406+0000","last_peered":"2024-09-10T20:22:09.924406+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:09.924406+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35746976,"num_objects":4364,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":34917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4394","reported_seq":4444,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.933420+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:09.933420+0000","last_peered":"2024-09-10T20:22:09.933420+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:09.933420+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35995648,"num_objects":4394,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":35152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4350","reported_seq":4422,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.443374+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:09.443374+0000","last_peered":"2024-09-10T20:22:09.443374+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:09.443374+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35583136,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4394","reported_seq":4466,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.345140+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:09.345140+0000","last_peered":"2024-09-10T20:22:09.345140+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:09.345140+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35951776,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":35124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4326","reported_seq":8286,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:09.936061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:09.936061+0000","last_peered":"2024-09-10T20:22:09.936061+0000","last_clean":"2024-09-10T20:22:09.936061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:09.936061+0000","last_undegraded":"2024-09-10T20:22:09.936061+0000","last_fullsized":"2024-09-10T20:22:09.936061+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35347024,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":879,"num_write_kb":7032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4358","reported_seq":4443,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.318799+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:12.318799+0000","last_peered":"2024-09-10T20:22:12.318799+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:12.318799+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35681440,"num_objects":4356,"num_object_clones":0,"num_object_copies":17424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":34846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4324","reported_seq":4387,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.897653+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:09.897653+0000","last_peered":"2024-09-10T20:22:09.897653+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:09.897653+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4324,"log_dups_size":0,"ondisk_log_size":4324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35376880,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4344","reported_seq":4396,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.879304+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:09.879304+0000","last_peered":"2024-09-10T20:22:09.879304+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:09.879304+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4344,"log_dups_size":0,"ondisk_log_size":4344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35576400,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2365}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4257","reported_seq":4340,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.262698+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:12.262698+0000","last_peered":"2024-09-10T20:22:12.262698+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:12.262698+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34863696,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":34047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4362","reported_seq":4421,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.805635+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:09.805635+0000","last_peered":"2024-09-10T20:22:09.805635+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:09.805635+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35697824,"num_objects":4358,"num_object_clones":0,"num_object_copies":17432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":34876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4529","reported_seq":4616,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.311587+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:12.311587+0000","last_peered":"2024-09-10T20:22:12.311587+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:12.311587+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37074080,"num_objects":4526,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":36213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2564}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4329","reported_seq":4401,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:09.447610+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:09.447610+0000","last_peered":"2024-09-10T20:22:09.447610+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:09.447610+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419296,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":34604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":572702000,"num_objects":69914,"num_object_clones":0,"num_object_copies":279656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62945,"num_write_kb":503469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6070272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038667776,"data_stored":1038667776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69973,"ondisk_log_size":69973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494548,"kb_used_data":278212,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93877292,"statfs":{"total":96636764160,"available":96130347008,"internally_reserved":0,"allocated":284889088,"data_stored":284694715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496468,"kb_used_data":280644,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93875372,"statfs":{"total":96636764160,"available":96128380928,"internally_reserved":0,"allocated":287379456,"data_stored":287185083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337284,"kb_used_data":175092,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94034556,"statfs":{"total":96636764160,"available":96291385344,"internally_reserved":0,"allocated":179294208,"data_stored":179099835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490220,"kb_used_data":278108,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881620,"statfs":{"total":96636764160,"available":96134778880,"internally_reserved":0,"allocated":284782592,"data_stored":284588219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284741632,"data_stored":284741632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179994624,"data_stored":179994624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":288407552,"data_stored":288407552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285523968,"data_stored":285523968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:15.835 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:16.471 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:16.473 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:16.493 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-10T20:22:15.927402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581729584,"num_objects":71016,"num_object_clones":0,"num_object_copies":284064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64047,"num_write_kb":512285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":862,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71075,"ondisk_log_size":71075,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1875944,"kb_used_data":1025064,"kb_used_omap":14,"kb_used_meta":850801,"kb_avail":375611416,"statfs":{"total":386547056640,"available":384626089984,"internally_reserved":0,"allocated":1049665536,"data_stored":1048888044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":871220790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,4,7,15,11,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":1998848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003931"},"pg_stats":[{"pgid":"1.f","version":"36'4483","reported_seq":4533,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.876293+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:14.876293+0000","last_peered":"2024-09-10T20:22:14.876293+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:14.876293+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36724736,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4462","reported_seq":4584,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:13.049544+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:13.049544+0000","last_peered":"2024-09-10T20:22:13.049544+0000","last_clean":"2024-09-10T20:22:13.049544+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:13.049544+0000","last_undegraded":"2024-09-10T20:22:13.049544+0000","last_fullsized":"2024-09-10T20:22:13.049544+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503552,"num_objects":4456,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":903,"num_write_kb":7224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4395","reported_seq":4479,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.292105+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:12.292105+0000","last_peered":"2024-09-10T20:22:12.292105+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:12.292105+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35977808,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":35142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4528","reported_seq":6280,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.438717+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:14.438717+0000","last_peered":"2024-09-10T20:22:14.438717+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:14.438717+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057696,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":842,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4458","reported_seq":4518,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.689466+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:14.689466+0000","last_peered":"2024-09-10T20:22:14.689466+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:14.689466+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36492448,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4456,"num_write_kb":35645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4484","reported_seq":4534,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.748944+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:14.748944+0000","last_peered":"2024-09-10T20:22:14.748944+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:14.748944+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36732928,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":35872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4451","reported_seq":4524,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.413092+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:14.413092+0000","last_peered":"2024-09-10T20:22:14.413092+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:14.413092+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4451,"log_dups_size":0,"ondisk_log_size":4451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36410528,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4449,"num_write_kb":35572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2492}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4494","reported_seq":4566,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.451667+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:14.451667+0000","last_peered":"2024-09-10T20:22:14.451667+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:14.451667+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36770976,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4428","reported_seq":8390,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:14.883913+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:14.883913+0000","last_peered":"2024-09-10T20:22:14.883913+0000","last_clean":"2024-09-10T20:22:14.883913+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:14.883913+0000","last_undegraded":"2024-09-10T20:22:14.883913+0000","last_fullsized":"2024-09-10T20:22:14.883913+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36182608,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":981,"num_write_kb":7848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4358","reported_seq":4443,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.318799+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:12.318799+0000","last_peered":"2024-09-10T20:22:12.318799+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:12.318799+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35681440,"num_objects":4356,"num_object_clones":0,"num_object_copies":17424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":34846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4439","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.873040+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:14.873040+0000","last_peered":"2024-09-10T20:22:14.873040+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:14.873040+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36318960,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":35476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4426","reported_seq":4478,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.877972+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:14.877972+0000","last_peered":"2024-09-10T20:22:14.877972+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:14.877972+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36248144,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":35399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4257","reported_seq":4340,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.262698+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:12.262698+0000","last_peered":"2024-09-10T20:22:12.262698+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:12.262698+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34863696,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":34047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4457","reported_seq":4516,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.739589+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:14.739589+0000","last_peered":"2024-09-10T20:22:14.739589+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:14.739589+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36476064,"num_objects":4453,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4455,"num_write_kb":35636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2478}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4529","reported_seq":4616,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.311587+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:12.311587+0000","last_peered":"2024-09-10T20:22:12.311587+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:12.311587+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37074080,"num_objects":4526,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":36213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2564}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4426","reported_seq":4497,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.353034+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:14.353034+0000","last_peered":"2024-09-10T20:22:14.353034+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:14.353034+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213920,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4424,"num_write_kb":35380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":581729584,"num_objects":71016,"num_object_clones":0,"num_object_copies":284064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64047,"num_write_kb":512285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":862,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1052086272,"data_stored":1052086272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71075,"ondisk_log_size":71075,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523292,"kb_used_data":284748,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93848548,"statfs":{"total":96636764160,"available":96100913152,"internally_reserved":0,"allocated":291581952,"data_stored":291387579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":244248272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496468,"kb_used_data":280644,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93875372,"statfs":{"total":96636764160,"available":96128380928,"internally_reserved":0,"allocated":287379456,"data_stored":287185083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337284,"kb_used_data":175092,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94034556,"statfs":{"total":96636764160,"available":96291385344,"internally_reserved":0,"allocated":179294208,"data_stored":179099835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518900,"kb_used_data":284580,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93852940,"statfs":{"total":96636764160,"available":96105410560,"internally_reserved":0,"allocated":291409920,"data_stored":291215547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,6,6,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291454976,"data_stored":291454976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179994624,"data_stored":179994624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":288407552,"data_stored":288407552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292229120,"data_stored":292229120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:16.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:16.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4483", 'reported_seq': 4533, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.876293+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:14.876293+0000', 'last_peered': '2024-09-10T20:22:14.876293+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:14.876293+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4483, 'log_dups_size': 0, 'ondisk_log_size': 4483, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36724736, 'num_objects': 4483, 'num_object_clones': 0, 'num_object_copies': 17932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4483, 'num_write_kb': 35864, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2517}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:16.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4395", 'reported_seq': 4479, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:12.292105+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:12.292105+0000', 'last_peered': '2024-09-10T20:22:12.292105+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:12.292105+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4395, 'log_dups_size': 0, 'ondisk_log_size': 4395, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35977808, 'num_objects': 4392, 'num_object_clones': 0, 'num_object_copies': 17568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4394, 'num_write_kb': 35142, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2430}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4528", 'reported_seq': 6280, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.438717+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:14.438717+0000', 'last_peered': '2024-09-10T20:22:14.438717+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:14.438717+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4528, 'log_dups_size': 0, 'ondisk_log_size': 4528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37057696, 'num_objects': 4524, 'num_object_clones': 0, 'num_object_copies': 18096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1163, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4526, 'num_write_kb': 36197, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 842, 'num_bytes_recovered': 6881280, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3361}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4458", 'reported_seq': 4518, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.689466+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:14.689466+0000', 'last_peered': '2024-09-10T20:22:14.689466+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:14.689466+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4458, 'log_dups_size': 0, 'ondisk_log_size': 4458, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36492448, 'num_objects': 4455, 'num_object_clones': 0, 'num_object_copies': 17820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4456, 'num_write_kb': 35645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2475}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4484", 'reported_seq': 4534, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.748944+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:14.748944+0000', 'last_peered': '2024-09-10T20:22:14.748944+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:14.748944+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4484, 'log_dups_size': 0, 'ondisk_log_size': 4484, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36732928, 'num_objects': 4484, 'num_object_clones': 0, 'num_object_copies': 17936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4484, 'num_write_kb': 35872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2546}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4451", 'reported_seq': 4524, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.413092+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:14.413092+0000', 'last_peered': '2024-09-10T20:22:14.413092+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:14.413092+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4451, 'log_dups_size': 0, 'ondisk_log_size': 4451, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36410528, 'num_objects': 4445, 'num_object_clones': 0, 'num_object_copies': 17780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4449, 'num_write_kb': 35572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2492}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:16.495 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4494", 'reported_seq': 4566, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.451667+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:14.451667+0000', 'last_peered': '2024-09-10T20:22:14.451667+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:14.451667+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4494, 'log_dups_size': 0, 'ondisk_log_size': 4494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36770976, 'num_objects': 4489, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4492, 'num_write_kb': 35924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4358", 'reported_seq': 4443, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:12.318799+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:12.318799+0000', 'last_peered': '2024-09-10T20:22:12.318799+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:12.318799+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4358, 'log_dups_size': 0, 'ondisk_log_size': 4358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35681440, 'num_objects': 4356, 'num_object_clones': 0, 'num_object_copies': 17424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4356, 'num_write_kb': 34846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2453}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4439", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.873040+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:14.873040+0000', 'last_peered': '2024-09-10T20:22:14.873040+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:14.873040+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4439, 'log_dups_size': 0, 'ondisk_log_size': 4439, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36318960, 'num_objects': 4434, 'num_object_clones': 0, 'num_object_copies': 17736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4436, 'num_write_kb': 35476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2478}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4426", 'reported_seq': 4478, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.877972+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:14.877972+0000', 'last_peered': '2024-09-10T20:22:14.877972+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:14.877972+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4426, 'log_dups_size': 0, 'ondisk_log_size': 4426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36248144, 'num_objects': 4425, 'num_object_clones': 0, 'num_object_copies': 17700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4425, 'num_write_kb': 35399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2447}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4257", 'reported_seq': 4340, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:12.262698+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:12.262698+0000', 'last_peered': '2024-09-10T20:22:12.262698+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:12.262698+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4257, 'log_dups_size': 0, 'ondisk_log_size': 4257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34863696, 'num_objects': 4256, 'num_object_clones': 0, 'num_object_copies': 17024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4256, 'num_write_kb': 34047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2308}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:16.496 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4457", 'reported_seq': 4516, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.739589+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:14.739589+0000', 'last_peered': '2024-09-10T20:22:14.739589+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:14.739589+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4457, 'log_dups_size': 0, 'ondisk_log_size': 4457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36476064, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 17812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4455, 'num_write_kb': 35636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:16.497 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:16.497 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4529", 'reported_seq': 4616, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:12.311587+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:12.311587+0000', 'last_peered': '2024-09-10T20:22:12.311587+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:12.311587+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4529, 'log_dups_size': 0, 'ondisk_log_size': 4529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37074080, 'num_objects': 4526, 'num_object_clones': 0, 'num_object_copies': 18104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4527, 'num_write_kb': 36213, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2564}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:16.497 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:16.497 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4426", 'reported_seq': 4497, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:14.353034+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:14.353034+0000', 'last_peered': '2024-09-10T20:22:14.353034+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:14.353034+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4426, 'log_dups_size': 0, 'ondisk_log_size': 4426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36213920, 'num_objects': 4421, 'num_object_clones': 0, 'num_object_copies': 17684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4424, 'num_write_kb': 35380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2442}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:16.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:16.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 155 20 71811 71791 3.61803 2.17969 0.0672833 0.0431729 2024-09-10T20:22:17.094 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:17.095 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26726/284064 objects degraded (9.408%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":225,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71016,"data_bytes":581729584,"bytes_used":1920966656,"bytes_avail":384626089984,"bytes_total":386547056640,"degraded_objects":26726,"degraded_total":284064,"degraded_ratio":0.094084431677368491,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":166516,"recovering_keys_per_sec":0,"num_objects_recovered":244,"num_bytes_recovered":1998848,"num_keys_recovered":0,"write_bytes_sec":3015033,"read_op_per_sec":0,"write_op_per_sec":368},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:17.128 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:17.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:17.684 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:17.685 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-10T20:22:15.927402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581729584,"num_objects":71016,"num_object_clones":0,"num_object_copies":284064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64047,"num_write_kb":512285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":862,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71075,"ondisk_log_size":71075,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1875944,"kb_used_data":1025064,"kb_used_omap":14,"kb_used_meta":850801,"kb_avail":375611416,"statfs":{"total":386547056640,"available":384626089984,"internally_reserved":0,"allocated":1049665536,"data_stored":1048888044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":871220790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,4,7,15,11,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":1998848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003931"},"pg_stats":[{"pgid":"1.f","version":"36'4483","reported_seq":4533,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.876293+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:14.876293+0000","last_peered":"2024-09-10T20:22:14.876293+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:14.876293+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36724736,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4462","reported_seq":4584,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:13.049544+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:13.049544+0000","last_peered":"2024-09-10T20:22:13.049544+0000","last_clean":"2024-09-10T20:22:13.049544+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:13.049544+0000","last_undegraded":"2024-09-10T20:22:13.049544+0000","last_fullsized":"2024-09-10T20:22:13.049544+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503552,"num_objects":4456,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":903,"num_write_kb":7224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4395","reported_seq":4479,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.292105+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:12.292105+0000","last_peered":"2024-09-10T20:22:12.292105+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:12.292105+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35977808,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":35142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4528","reported_seq":6280,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.438717+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:14.438717+0000","last_peered":"2024-09-10T20:22:14.438717+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:14.438717+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057696,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":842,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4458","reported_seq":4518,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.689466+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:14.689466+0000","last_peered":"2024-09-10T20:22:14.689466+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:14.689466+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36492448,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4456,"num_write_kb":35645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4484","reported_seq":4534,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.748944+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:14.748944+0000","last_peered":"2024-09-10T20:22:14.748944+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:14.748944+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36732928,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":35872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4451","reported_seq":4524,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.413092+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:14.413092+0000","last_peered":"2024-09-10T20:22:14.413092+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:14.413092+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4451,"log_dups_size":0,"ondisk_log_size":4451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36410528,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4449,"num_write_kb":35572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2492}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4494","reported_seq":4566,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.451667+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:14.451667+0000","last_peered":"2024-09-10T20:22:14.451667+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:14.451667+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36770976,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4428","reported_seq":8390,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:14.883913+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:14.883913+0000","last_peered":"2024-09-10T20:22:14.883913+0000","last_clean":"2024-09-10T20:22:14.883913+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:14.883913+0000","last_undegraded":"2024-09-10T20:22:14.883913+0000","last_fullsized":"2024-09-10T20:22:14.883913+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36182608,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":981,"num_write_kb":7848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4358","reported_seq":4443,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.318799+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:12.318799+0000","last_peered":"2024-09-10T20:22:12.318799+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:12.318799+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35681440,"num_objects":4356,"num_object_clones":0,"num_object_copies":17424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":34846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4439","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.873040+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:14.873040+0000","last_peered":"2024-09-10T20:22:14.873040+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:14.873040+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36318960,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":35476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4426","reported_seq":4478,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.877972+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:14.877972+0000","last_peered":"2024-09-10T20:22:14.877972+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:14.877972+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36248144,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":35399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4257","reported_seq":4340,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.262698+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:12.262698+0000","last_peered":"2024-09-10T20:22:12.262698+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:12.262698+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34863696,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":34047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4457","reported_seq":4516,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.739589+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:14.739589+0000","last_peered":"2024-09-10T20:22:14.739589+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:14.739589+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36476064,"num_objects":4453,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4455,"num_write_kb":35636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2478}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4529","reported_seq":4616,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:12.311587+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:12.311587+0000","last_peered":"2024-09-10T20:22:12.311587+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:12.311587+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37074080,"num_objects":4526,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":36213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2564}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4426","reported_seq":4497,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.353034+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:14.353034+0000","last_peered":"2024-09-10T20:22:14.353034+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:14.353034+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213920,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4424,"num_write_kb":35380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":581729584,"num_objects":71016,"num_object_clones":0,"num_object_copies":284064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64047,"num_write_kb":512285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":862,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1052086272,"data_stored":1052086272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71075,"ondisk_log_size":71075,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523292,"kb_used_data":284748,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93848548,"statfs":{"total":96636764160,"available":96100913152,"internally_reserved":0,"allocated":291581952,"data_stored":291387579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":244248272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496468,"kb_used_data":280644,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93875372,"statfs":{"total":96636764160,"available":96128380928,"internally_reserved":0,"allocated":287379456,"data_stored":287185083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337284,"kb_used_data":175092,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94034556,"statfs":{"total":96636764160,"available":96291385344,"internally_reserved":0,"allocated":179294208,"data_stored":179099835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518900,"kb_used_data":284580,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93852940,"statfs":{"total":96636764160,"available":96105410560,"internally_reserved":0,"allocated":291409920,"data_stored":291215547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,6,6,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291454976,"data_stored":291454976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179994624,"data_stored":179994624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":288407552,"data_stored":288407552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292229120,"data_stored":292229120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:17.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 156 20 72093 72073 3.60896 2.20312 0.0288506 0.0432702 2024-09-10T20:22:18.793 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 157 20 72451 72431 3.60379 2.79688 0.0572927 0.0433433 2024-09-10T20:22:19.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 158 20 72799 72779 3.59819 2.71875 0.0306949 0.04341 2024-09-10T20:22:20.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:20.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 159 20 73146 73126 3.5926 2.71094 0.0596517 0.0434773 2024-09-10T20:22:21.297 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:21.322 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-10T20:22:19.928705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589045040,"num_objects":71909,"num_object_clones":0,"num_object_copies":287636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64940,"num_write_kb":519429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71968,"ondisk_log_size":71968,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1943912,"kb_used_data":1044456,"kb_used_omap":14,"kb_used_meta":899377,"kb_avail":375543448,"statfs":{"total":386547056640,"available":384556490752,"internally_reserved":0,"allocated":1069522944,"data_stored":1068745452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":920962614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,10,14,13,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30834688,"num_objects":3764,"num_object_clones":0,"num_object_copies":15056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":30112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":2465792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003881"},"pg_stats":[{"pgid":"1.f","version":"36'4483","reported_seq":4533,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.876293+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:14.876293+0000","last_peered":"2024-09-10T20:22:14.876293+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:14.876293+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36724736,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4558","reported_seq":4680,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:18.076115+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:18.076115+0000","last_peered":"2024-09-10T20:22:18.076115+0000","last_clean":"2024-09-10T20:22:18.076115+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:18.076115+0000","last_undegraded":"2024-09-10T20:22:18.076115+0000","last_fullsized":"2024-09-10T20:22:18.076115+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37289984,"num_objects":4552,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":999,"num_write_kb":7992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4491","reported_seq":4575,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.286686+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:17.286686+0000","last_peered":"2024-09-10T20:22:17.286686+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:17.286686+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36764240,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":35910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4628","reported_seq":6626,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.498592+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:19.498592+0000","last_peered":"2024-09-10T20:22:19.498592+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:19.498592+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876896,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4626,"num_write_kb":36997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":965,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3582}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4458","reported_seq":4518,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.689466+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:14.689466+0000","last_peered":"2024-09-10T20:22:14.689466+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:14.689466+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36492448,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4456,"num_write_kb":35645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4484","reported_seq":4534,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.748944+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:14.748944+0000","last_peered":"2024-09-10T20:22:14.748944+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:14.748944+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36732928,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":35872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4556","reported_seq":4627,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.533992+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:19.533992+0000","last_peered":"2024-09-10T20:22:19.533992+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:19.533992+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270688,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":36412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4587","reported_seq":4659,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.516612+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:19.516612+0000","last_peered":"2024-09-10T20:22:19.516612+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:19.516612+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37532832,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":36668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4428","reported_seq":8390,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:14.883913+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:14.883913+0000","last_peered":"2024-09-10T20:22:14.883913+0000","last_clean":"2024-09-10T20:22:14.883913+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:14.883913+0000","last_undegraded":"2024-09-10T20:22:14.883913+0000","last_fullsized":"2024-09-10T20:22:14.883913+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36182608,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":981,"num_write_kb":7848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4460","reported_seq":4545,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.253262+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:17.253262+0000","last_peered":"2024-09-10T20:22:17.253262+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:17.253262+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36517024,"num_objects":4458,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4458,"num_write_kb":35662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4439","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.873040+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:14.873040+0000","last_peered":"2024-09-10T20:22:14.873040+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:14.873040+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36318960,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":35476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4426","reported_seq":4478,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.877972+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:14.877972+0000","last_peered":"2024-09-10T20:22:14.877972+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:14.877972+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36248144,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":35399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4362","reported_seq":4445,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.247715+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:17.247715+0000","last_peered":"2024-09-10T20:22:17.247715+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:17.247715+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35723856,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4361,"num_write_kb":34887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4457","reported_seq":4516,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:14.739589+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:14.739589+0000","last_peered":"2024-09-10T20:22:14.739589+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:14.739589+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36476064,"num_objects":4453,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4455,"num_write_kb":35636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2478}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4628","reported_seq":4715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.256396+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:17.256396+0000","last_peered":"2024-09-10T20:22:17.256396+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:17.256396+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37885088,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4626,"num_write_kb":37005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4523","reported_seq":4593,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.521320+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:19.521320+0000","last_peered":"2024-09-10T20:22:19.521320+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:19.521320+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37008544,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":36156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":589045040,"num_objects":71909,"num_object_clones":0,"num_object_copies":287636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64940,"num_write_kb":519429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1071845376,"data_stored":1071845376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71968,"ondisk_log_size":71968,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523292,"kb_used_data":284748,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93848548,"statfs":{"total":96636764160,"available":96100913152,"internally_reserved":0,"allocated":291581952,"data_stored":291387579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":244248272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525180,"kb_used_data":287148,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93846660,"statfs":{"total":96636764160,"available":96098979840,"internally_reserved":0,"allocated":294039552,"data_stored":293845179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366232,"kb_used_data":181832,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":94005608,"statfs":{"total":96636764160,"available":96261742592,"internally_reserved":0,"allocated":186195968,"data_stored":186001595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":188804818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529208,"kb_used_data":290728,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93842632,"statfs":{"total":96636764160,"available":96094855168,"internally_reserved":0,"allocated":297705472,"data_stored":297511099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,13,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297881600,"data_stored":297881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186892288,"data_stored":186892288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294842368,"data_stored":294842368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292229120,"data_stored":292229120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:21.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:21.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:22:21.787271+0000 min lat: 0.00230373 max lat: 12.2408 avg lat: 0.0435705 2024-09-10T20:22:21.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:22:21.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 160 20 73449 73429 3.58494 2.36719 0.083456 0.0435705 2024-09-10T20:22:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:21.955 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:21.976 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-10T20:22:21.929297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594459952,"num_objects":72570,"num_object_clones":0,"num_object_copies":290280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65601,"num_write_kb":524717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72629,"ondisk_log_size":72629,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1954316,"kb_used_data":1050700,"kb_used_omap":14,"kb_used_meta":903537,"kb_avail":375533044,"statfs":{"total":386547056640,"available":384545837056,"internally_reserved":0,"allocated":1075916800,"data_stored":1075139308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":925222454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,15,16,15,17,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32022528,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003767"},"pg_stats":[{"pgid":"1.f","version":"36'4585","reported_seq":4635,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.930497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:19.930497+0000","last_peered":"2024-09-10T20:22:19.930497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:19.930497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37560320,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":36680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4558","reported_seq":4680,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:18.076115+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:18.076115+0000","last_peered":"2024-09-10T20:22:18.076115+0000","last_clean":"2024-09-10T20:22:18.076115+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:18.076115+0000","last_undegraded":"2024-09-10T20:22:18.076115+0000","last_fullsized":"2024-09-10T20:22:18.076115+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37289984,"num_objects":4552,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":999,"num_write_kb":7992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4491","reported_seq":4575,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.286686+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:17.286686+0000","last_peered":"2024-09-10T20:22:17.286686+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:17.286686+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36764240,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":35910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4628","reported_seq":6626,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.498592+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:19.498592+0000","last_peered":"2024-09-10T20:22:19.498592+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:19.498592+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876896,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4626,"num_write_kb":36997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":965,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3582}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4547","reported_seq":4606,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.940470+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:19.940470+0000","last_peered":"2024-09-10T20:22:19.940470+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:19.940470+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37221536,"num_objects":4544,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4545,"num_write_kb":36357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4581","reported_seq":4630,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.943505+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:19.943505+0000","last_peered":"2024-09-10T20:22:19.943505+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:19.943505+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":36648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4556","reported_seq":4627,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.533992+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:19.533992+0000","last_peered":"2024-09-10T20:22:19.533992+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:19.533992+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270688,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":36412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4587","reported_seq":4659,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.516612+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:19.516612+0000","last_peered":"2024-09-10T20:22:19.516612+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:19.516612+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37532832,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":36668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4527","reported_seq":8489,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:19.942163+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:19.942163+0000","last_peered":"2024-09-10T20:22:19.942163+0000","last_clean":"2024-09-10T20:22:19.942163+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:19.942163+0000","last_undegraded":"2024-09-10T20:22:19.942163+0000","last_fullsized":"2024-09-10T20:22:19.942163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36993616,"num_objects":4516,"num_object_clones":0,"num_object_copies":18064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1080,"num_write_kb":8640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4460","reported_seq":4545,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.253262+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:17.253262+0000","last_peered":"2024-09-10T20:22:17.253262+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:17.253262+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36517024,"num_objects":4458,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4458,"num_write_kb":35662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4520","reported_seq":4583,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.792848+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:19.792848+0000","last_peered":"2024-09-10T20:22:19.792848+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:19.792848+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4520,"log_dups_size":0,"ondisk_log_size":4520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36982512,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4517,"num_write_kb":36124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4526","reported_seq":4578,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.917551+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:19.917551+0000","last_peered":"2024-09-10T20:22:19.917551+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:19.917551+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37067344,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4525,"num_write_kb":36199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4362","reported_seq":4445,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.247715+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:17.247715+0000","last_peered":"2024-09-10T20:22:17.247715+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:17.247715+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35723856,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4361,"num_write_kb":34887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4550","reported_seq":4609,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.988471+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:19.988471+0000","last_peered":"2024-09-10T20:22:19.988471+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:19.988471+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37237920,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":36380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4628","reported_seq":4715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.256396+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:17.256396+0000","last_peered":"2024-09-10T20:22:17.256396+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:17.256396+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37885088,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4626,"num_write_kb":37005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4523","reported_seq":4593,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.521320+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:19.521320+0000","last_peered":"2024-09-10T20:22:19.521320+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:19.521320+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37008544,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":36156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594459952,"num_objects":72570,"num_object_clones":0,"num_object_copies":290280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65601,"num_write_kb":524717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1078099968,"data_stored":1078099968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72629,"ondisk_log_size":72629,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533696,"kb_used_data":290992,"kb_used_omap":4,"kb_used_meta":242683,"kb_avail":93838144,"statfs":{"total":96636764160,"available":96090259456,"internally_reserved":0,"allocated":297975808,"data_stored":297781435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":248508112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525180,"kb_used_data":287148,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93846660,"statfs":{"total":96636764160,"available":96098979840,"internally_reserved":0,"allocated":294039552,"data_stored":293845179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366232,"kb_used_data":181832,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":94005608,"statfs":{"total":96636764160,"available":96261742592,"internally_reserved":0,"allocated":186195968,"data_stored":186001595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":188804818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529208,"kb_used_data":290728,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93842632,"statfs":{"total":96636764160,"available":96094855168,"internally_reserved":0,"allocated":297705472,"data_stored":297511099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,13,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297881600,"data_stored":297881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186892288,"data_stored":186892288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294842368,"data_stored":294842368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298483712,"data_stored":298483712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:21.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:21.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4585", 'reported_seq': 4635, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.930497+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:19.930497+0000', 'last_peered': '2024-09-10T20:22:19.930497+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:19.930497+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4585, 'log_dups_size': 0, 'ondisk_log_size': 4585, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37560320, 'num_objects': 4585, 'num_object_clones': 0, 'num_object_copies': 18340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4585, 'num_write_kb': 36680, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2619}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:21.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:21.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4491", 'reported_seq': 4575, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:17.286686+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:17.286686+0000', 'last_peered': '2024-09-10T20:22:17.286686+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:17.286686+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4491, 'log_dups_size': 0, 'ondisk_log_size': 4491, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36764240, 'num_objects': 4488, 'num_object_clones': 0, 'num_object_copies': 17952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 35910, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2526}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4628", 'reported_seq': 6626, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.498592+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:19.498592+0000', 'last_peered': '2024-09-10T20:22:19.498592+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:19.498592+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4628, 'log_dups_size': 0, 'ondisk_log_size': 4628, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37876896, 'num_objects': 4624, 'num_object_clones': 0, 'num_object_copies': 18496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1042, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4626, 'num_write_kb': 36997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 965, 'num_bytes_recovered': 7872512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3582}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4547", 'reported_seq': 4606, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.940470+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:19.940470+0000', 'last_peered': '2024-09-10T20:22:19.940470+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:19.940470+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4547, 'log_dups_size': 0, 'ondisk_log_size': 4547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37221536, 'num_objects': 4544, 'num_object_clones': 0, 'num_object_copies': 18176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4545, 'num_write_kb': 36357, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2564}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4581", 'reported_seq': 4630, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.943505+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:19.943505+0000', 'last_peered': '2024-09-10T20:22:19.943505+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:19.943505+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4581, 'log_dups_size': 0, 'ondisk_log_size': 4581, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37527552, 'num_objects': 4581, 'num_object_clones': 0, 'num_object_copies': 18324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4581, 'num_write_kb': 36648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2643}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4556", 'reported_seq': 4627, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.533992+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:19.533992+0000', 'last_peered': '2024-09-10T20:22:19.533992+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:19.533992+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4556, 'log_dups_size': 0, 'ondisk_log_size': 4556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37270688, 'num_objects': 4550, 'num_object_clones': 0, 'num_object_copies': 18200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4554, 'num_write_kb': 36412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2597}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4587", 'reported_seq': 4659, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.516612+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:19.516612+0000', 'last_peered': '2024-09-10T20:22:19.516612+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:19.516612+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4587, 'log_dups_size': 0, 'ondisk_log_size': 4587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37532832, 'num_objects': 4582, 'num_object_clones': 0, 'num_object_copies': 18328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4585, 'num_write_kb': 36668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:21.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4460", 'reported_seq': 4545, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:17.253262+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:17.253262+0000', 'last_peered': '2024-09-10T20:22:17.253262+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:17.253262+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4460, 'log_dups_size': 0, 'ondisk_log_size': 4460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36517024, 'num_objects': 4458, 'num_object_clones': 0, 'num_object_copies': 17832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4458, 'num_write_kb': 35662, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2555}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4520", 'reported_seq': 4583, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.792848+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:19.792848+0000', 'last_peered': '2024-09-10T20:22:19.792848+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:19.792848+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4520, 'log_dups_size': 0, 'ondisk_log_size': 4520, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36982512, 'num_objects': 4515, 'num_object_clones': 0, 'num_object_copies': 18060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4517, 'num_write_kb': 36124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2559}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4526", 'reported_seq': 4578, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.917551+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:19.917551+0000', 'last_peered': '2024-09-10T20:22:19.917551+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:19.917551+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4526, 'log_dups_size': 0, 'ondisk_log_size': 4526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37067344, 'num_objects': 4525, 'num_object_clones': 0, 'num_object_copies': 18100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4525, 'num_write_kb': 36199, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2547}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4362", 'reported_seq': 4445, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:17.247715+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:17.247715+0000', 'last_peered': '2024-09-10T20:22:17.247715+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:17.247715+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4362, 'log_dups_size': 0, 'ondisk_log_size': 4362, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35723856, 'num_objects': 4361, 'num_object_clones': 0, 'num_object_copies': 17444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4361, 'num_write_kb': 34887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4550", 'reported_seq': 4609, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.988471+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:19.988471+0000', 'last_peered': '2024-09-10T20:22:19.988471+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:19.988471+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4550, 'log_dups_size': 0, 'ondisk_log_size': 4550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37237920, 'num_objects': 4546, 'num_object_clones': 0, 'num_object_copies': 18184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 36380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2571}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4628", 'reported_seq': 4715, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:17.256396+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:17.256396+0000', 'last_peered': '2024-09-10T20:22:17.256396+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:17.256396+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4628, 'log_dups_size': 0, 'ondisk_log_size': 4628, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37885088, 'num_objects': 4625, 'num_object_clones': 0, 'num_object_copies': 18500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4626, 'num_write_kb': 37005, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:21.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:21.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4523", 'reported_seq': 4593, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:19.521320+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:19.521320+0000', 'last_peered': '2024-09-10T20:22:19.521320+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:19.521320+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4523, 'log_dups_size': 0, 'ondisk_log_size': 4523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37008544, 'num_objects': 4518, 'num_object_clones': 0, 'num_object_copies': 18072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4521, 'num_write_kb': 36156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:21.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:22.616 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:22.616 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26605/290280 objects degraded (9.165%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":231,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72570,"data_bytes":594459952,"bytes_used":2001219584,"bytes_avail":384545837056,"bytes_total":386547056640,"degraded_objects":26605,"degraded_total":290280,"degraded_ratio":0.091652886867851724,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":150139,"recovering_keys_per_sec":0,"num_objects_recovered":222,"num_bytes_recovered":1802240,"num_keys_recovered":0,"write_bytes_sec":2667706,"read_op_per_sec":0,"write_op_per_sec":325},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:22.641 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:22.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:22.794 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 161 20 73710 73690 3.57534 2.03906 0.0569101 0.0436784 2024-09-10T20:22:23.211 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:23.211 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:23.232 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-10T20:22:21.929297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594459952,"num_objects":72570,"num_object_clones":0,"num_object_copies":290280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65601,"num_write_kb":524717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72629,"ondisk_log_size":72629,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1954316,"kb_used_data":1050700,"kb_used_omap":14,"kb_used_meta":903537,"kb_avail":375533044,"statfs":{"total":386547056640,"available":384545837056,"internally_reserved":0,"allocated":1075916800,"data_stored":1075139308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":925222454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,15,16,15,17,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32022528,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003767"},"pg_stats":[{"pgid":"1.f","version":"36'4585","reported_seq":4635,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.930497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:19.930497+0000","last_peered":"2024-09-10T20:22:19.930497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:19.930497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37560320,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":36680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4558","reported_seq":4680,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:18.076115+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:18.076115+0000","last_peered":"2024-09-10T20:22:18.076115+0000","last_clean":"2024-09-10T20:22:18.076115+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:18.076115+0000","last_undegraded":"2024-09-10T20:22:18.076115+0000","last_fullsized":"2024-09-10T20:22:18.076115+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37289984,"num_objects":4552,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":999,"num_write_kb":7992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4491","reported_seq":4575,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.286686+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:17.286686+0000","last_peered":"2024-09-10T20:22:17.286686+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:17.286686+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36764240,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":35910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4628","reported_seq":6626,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.498592+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:19.498592+0000","last_peered":"2024-09-10T20:22:19.498592+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:19.498592+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876896,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4626,"num_write_kb":36997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":965,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3582}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4547","reported_seq":4606,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.940470+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:19.940470+0000","last_peered":"2024-09-10T20:22:19.940470+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:19.940470+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37221536,"num_objects":4544,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4545,"num_write_kb":36357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4581","reported_seq":4630,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.943505+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:19.943505+0000","last_peered":"2024-09-10T20:22:19.943505+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:19.943505+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":36648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4556","reported_seq":4627,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.533992+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:19.533992+0000","last_peered":"2024-09-10T20:22:19.533992+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:19.533992+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270688,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":36412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4587","reported_seq":4659,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.516612+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:19.516612+0000","last_peered":"2024-09-10T20:22:19.516612+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:19.516612+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37532832,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":36668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4527","reported_seq":8489,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:19.942163+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:19.942163+0000","last_peered":"2024-09-10T20:22:19.942163+0000","last_clean":"2024-09-10T20:22:19.942163+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:19.942163+0000","last_undegraded":"2024-09-10T20:22:19.942163+0000","last_fullsized":"2024-09-10T20:22:19.942163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36993616,"num_objects":4516,"num_object_clones":0,"num_object_copies":18064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1080,"num_write_kb":8640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4460","reported_seq":4545,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.253262+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:17.253262+0000","last_peered":"2024-09-10T20:22:17.253262+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:17.253262+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36517024,"num_objects":4458,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4458,"num_write_kb":35662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4520","reported_seq":4583,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.792848+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:19.792848+0000","last_peered":"2024-09-10T20:22:19.792848+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:19.792848+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4520,"log_dups_size":0,"ondisk_log_size":4520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36982512,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4517,"num_write_kb":36124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4526","reported_seq":4578,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.917551+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:19.917551+0000","last_peered":"2024-09-10T20:22:19.917551+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:19.917551+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37067344,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4525,"num_write_kb":36199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4362","reported_seq":4445,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.247715+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:17.247715+0000","last_peered":"2024-09-10T20:22:17.247715+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:17.247715+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35723856,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4361,"num_write_kb":34887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4550","reported_seq":4609,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.988471+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:19.988471+0000","last_peered":"2024-09-10T20:22:19.988471+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:19.988471+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37237920,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":36380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4628","reported_seq":4715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:17.256396+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:17.256396+0000","last_peered":"2024-09-10T20:22:17.256396+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:17.256396+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37885088,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4626,"num_write_kb":37005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4523","reported_seq":4593,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:19.521320+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:19.521320+0000","last_peered":"2024-09-10T20:22:19.521320+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:19.521320+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37008544,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":36156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594459952,"num_objects":72570,"num_object_clones":0,"num_object_copies":290280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65601,"num_write_kb":524717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7872512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1078099968,"data_stored":1078099968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72629,"ondisk_log_size":72629,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533696,"kb_used_data":290992,"kb_used_omap":4,"kb_used_meta":242683,"kb_avail":93838144,"statfs":{"total":96636764160,"available":96090259456,"internally_reserved":0,"allocated":297975808,"data_stored":297781435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":248508112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525180,"kb_used_data":287148,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93846660,"statfs":{"total":96636764160,"available":96098979840,"internally_reserved":0,"allocated":294039552,"data_stored":293845179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366232,"kb_used_data":181832,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":94005608,"statfs":{"total":96636764160,"available":96261742592,"internally_reserved":0,"allocated":186195968,"data_stored":186001595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":188804818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529208,"kb_used_data":290728,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93842632,"statfs":{"total":96636764160,"available":96094855168,"internally_reserved":0,"allocated":297705472,"data_stored":297511099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,13,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297881600,"data_stored":297881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186892288,"data_stored":186892288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294842368,"data_stored":294842368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298483712,"data_stored":298483712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:23.794 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 162 20 74025 74005 3.56846 2.46094 0.0738286 0.0437739 2024-09-10T20:22:24.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 163 20 74309 74289 3.56018 2.21875 0.0489548 0.0438744 2024-09-10T20:22:25.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 164 20 74652 74632 3.55481 2.67969 0.0628772 0.0439431 2024-09-10T20:22:26.234 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:26.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 165 20 74972 74952 3.54842 2.5 0.0685625 0.0440171 2024-09-10T20:22:26.810 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:26.810 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:26.845 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-10T20:22:25.930532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":607214896,"num_objects":74127,"num_object_clones":0,"num_object_copies":296508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67158,"num_write_kb":537173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74186,"ondisk_log_size":74186,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1997588,"kb_used_data":1077268,"kb_used_omap":14,"kb_used_meta":920241,"kb_avail":375489772,"statfs":{"total":386547056640,"available":384501526528,"internally_reserved":0,"allocated":1103122432,"data_stored":1102344940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":942327350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,2,9,10,12,13,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34512896,"num_objects":4213,"num_object_clones":0,"num_object_copies":16852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4213,"num_write_kb":33704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2940928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003761"},"pg_stats":[{"pgid":"1.f","version":"36'4692","reported_seq":4742,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.903068+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:24.903068+0000","last_peered":"2024-09-10T20:22:24.903068+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:24.903068+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38436864,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4692,"num_write_kb":37536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4645","reported_seq":4767,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:23.027797+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:23.027797+0000","last_peered":"2024-09-10T20:22:23.027797+0000","last_clean":"2024-09-10T20:22:23.027797+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:23.027797+0000","last_undegraded":"2024-09-10T20:22:23.027797+0000","last_fullsized":"2024-09-10T20:22:23.027797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38002688,"num_objects":4639,"num_object_clones":0,"num_object_copies":18556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1086,"num_write_kb":8688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4592","reported_seq":4676,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.155913+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:22.155913+0000","last_peered":"2024-09-10T20:22:22.155913+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:22.155913+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37591632,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4711","reported_seq":6987,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.609424+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:24.609424+0000","last_peered":"2024-09-10T20:22:24.609424+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:24.609424+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38556832,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":37661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4641","reported_seq":4700,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.884228+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:24.884228+0000","last_peered":"2024-09-10T20:22:24.884228+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:24.884228+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37991584,"num_objects":4638,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":37109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4668","reported_seq":4718,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.979685+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:24.979685+0000","last_peered":"2024-09-10T20:22:24.979685+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:24.979685+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38240256,"num_objects":4668,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4668,"num_write_kb":37344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4636","reported_seq":4709,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:23.992624+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:23.992624+0000","last_peered":"2024-09-10T20:22:23.992624+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:23.992624+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926048,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4634,"num_write_kb":37052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2677}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4682","reported_seq":4753,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.591474+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:24.591474+0000","last_peered":"2024-09-10T20:22:24.591474+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:24.591474+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311072,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4680,"num_write_kb":37428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4609","reported_seq":8571,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:24.973873+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:24.973873+0000","last_peered":"2024-09-10T20:22:24.973873+0000","last_clean":"2024-09-10T20:22:24.973873+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:24.973873+0000","last_undegraded":"2024-09-10T20:22:24.973873+0000","last_fullsized":"2024-09-10T20:22:24.973873+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37665360,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1162,"num_write_kb":9296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4572","reported_seq":4657,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.116662+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:22.116662+0000","last_peered":"2024-09-10T20:22:22.116662+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:22.116662+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4572,"log_dups_size":0,"ondisk_log_size":4572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37434528,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":36558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4618","reported_seq":4681,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.984726+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:24.984726+0000","last_peered":"2024-09-10T20:22:24.984726+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:24.984726+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37785328,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4630","reported_seq":4682,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.989451+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:24.989451+0000","last_peered":"2024-09-10T20:22:24.989451+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:24.989451+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37919312,"num_objects":4629,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":37031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4474","reported_seq":4557,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.162784+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:22.162784+0000","last_peered":"2024-09-10T20:22:22.162784+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:22.162784+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36641360,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4473,"num_write_kb":35783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2525}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4646","reported_seq":4705,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.908022+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:24.908022+0000","last_peered":"2024-09-10T20:22:24.908022+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:24.908022+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4646,"log_dups_size":0,"ondisk_log_size":4646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38024352,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4644,"num_write_kb":37148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2667}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4751","reported_seq":4838,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.157979+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:22.157979+0000","last_peered":"2024-09-10T20:22:22.157979+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:22.157979+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38892704,"num_objects":4748,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2786}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4619","reported_seq":4691,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.569475+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:24.569475+0000","last_peered":"2024-09-10T20:22:24.569475+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:24.569475+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37794976,"num_objects":4614,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":36924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":607214896,"num_objects":74127,"num_object_clones":0,"num_object_copies":296508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67158,"num_write_kb":537173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1104216064,"data_stored":1104216064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74186,"ondisk_log_size":74186,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545328,"kb_used_data":297056,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93826512,"statfs":{"total":96636764160,"available":96078348288,"internally_reserved":0,"allocated":304185344,"data_stored":303990971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542424,"kb_used_data":294664,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93829416,"statfs":{"total":96636764160,"available":96081321984,"internally_reserved":0,"allocated":301735936,"data_stored":301541563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364812,"kb_used_data":188732,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94007028,"statfs":{"total":96636764160,"available":96263196672,"internally_reserved":0,"allocated":193261568,"data_stored":193067195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":180285138},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,1,0,0,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545024,"kb_used_data":296816,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93826816,"statfs":{"total":96636764160,"available":96078659584,"internally_reserved":0,"allocated":303939584,"data_stored":303745211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304107520,"data_stored":304107520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193880064,"data_stored":193880064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301469696,"data_stored":301469696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304758784,"data_stored":304758784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:26.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:27.442 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:27.473 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-10T20:22:25.930532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":607214896,"num_objects":74127,"num_object_clones":0,"num_object_copies":296508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67158,"num_write_kb":537173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74186,"ondisk_log_size":74186,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1997588,"kb_used_data":1077268,"kb_used_omap":14,"kb_used_meta":920241,"kb_avail":375489772,"statfs":{"total":386547056640,"available":384501526528,"internally_reserved":0,"allocated":1103122432,"data_stored":1102344940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":942327350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,2,9,10,12,13,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34512896,"num_objects":4213,"num_object_clones":0,"num_object_copies":16852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4213,"num_write_kb":33704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2940928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003761"},"pg_stats":[{"pgid":"1.f","version":"36'4692","reported_seq":4742,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.903068+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:24.903068+0000","last_peered":"2024-09-10T20:22:24.903068+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:24.903068+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38436864,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4692,"num_write_kb":37536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4645","reported_seq":4767,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:23.027797+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:23.027797+0000","last_peered":"2024-09-10T20:22:23.027797+0000","last_clean":"2024-09-10T20:22:23.027797+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:23.027797+0000","last_undegraded":"2024-09-10T20:22:23.027797+0000","last_fullsized":"2024-09-10T20:22:23.027797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38002688,"num_objects":4639,"num_object_clones":0,"num_object_copies":18556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1086,"num_write_kb":8688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4592","reported_seq":4676,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.155913+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:22.155913+0000","last_peered":"2024-09-10T20:22:22.155913+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:22.155913+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37591632,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4711","reported_seq":6987,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.609424+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:24.609424+0000","last_peered":"2024-09-10T20:22:24.609424+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:24.609424+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38556832,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":37661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4641","reported_seq":4700,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.884228+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:24.884228+0000","last_peered":"2024-09-10T20:22:24.884228+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:24.884228+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37991584,"num_objects":4638,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":37109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4668","reported_seq":4718,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.979685+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:24.979685+0000","last_peered":"2024-09-10T20:22:24.979685+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:24.979685+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38240256,"num_objects":4668,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4668,"num_write_kb":37344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4636","reported_seq":4709,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:23.992624+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:23.992624+0000","last_peered":"2024-09-10T20:22:23.992624+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:23.992624+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926048,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4634,"num_write_kb":37052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2677}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4682","reported_seq":4753,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.591474+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:24.591474+0000","last_peered":"2024-09-10T20:22:24.591474+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:24.591474+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311072,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4680,"num_write_kb":37428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4609","reported_seq":8571,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:24.973873+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:24.973873+0000","last_peered":"2024-09-10T20:22:24.973873+0000","last_clean":"2024-09-10T20:22:24.973873+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:24.973873+0000","last_undegraded":"2024-09-10T20:22:24.973873+0000","last_fullsized":"2024-09-10T20:22:24.973873+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37665360,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1162,"num_write_kb":9296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4572","reported_seq":4657,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.116662+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:22.116662+0000","last_peered":"2024-09-10T20:22:22.116662+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:22.116662+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4572,"log_dups_size":0,"ondisk_log_size":4572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37434528,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":36558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4618","reported_seq":4681,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.984726+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:24.984726+0000","last_peered":"2024-09-10T20:22:24.984726+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:24.984726+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37785328,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4630","reported_seq":4682,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.989451+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:24.989451+0000","last_peered":"2024-09-10T20:22:24.989451+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:24.989451+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37919312,"num_objects":4629,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":37031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4474","reported_seq":4557,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.162784+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:22.162784+0000","last_peered":"2024-09-10T20:22:22.162784+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:22.162784+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36641360,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4473,"num_write_kb":35783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2525}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4646","reported_seq":4705,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.908022+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:24.908022+0000","last_peered":"2024-09-10T20:22:24.908022+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:24.908022+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4646,"log_dups_size":0,"ondisk_log_size":4646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38024352,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4644,"num_write_kb":37148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2667}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4751","reported_seq":4838,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:22.157979+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:22.157979+0000","last_peered":"2024-09-10T20:22:22.157979+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:22.157979+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38892704,"num_objects":4748,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2786}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4619","reported_seq":4691,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.569475+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:24.569475+0000","last_peered":"2024-09-10T20:22:24.569475+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:24.569475+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37794976,"num_objects":4614,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":36924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":607214896,"num_objects":74127,"num_object_clones":0,"num_object_copies":296508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67158,"num_write_kb":537173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1104216064,"data_stored":1104216064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74186,"ondisk_log_size":74186,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545328,"kb_used_data":297056,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93826512,"statfs":{"total":96636764160,"available":96078348288,"internally_reserved":0,"allocated":304185344,"data_stored":303990971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542424,"kb_used_data":294664,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93829416,"statfs":{"total":96636764160,"available":96081321984,"internally_reserved":0,"allocated":301735936,"data_stored":301541563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364812,"kb_used_data":188732,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94007028,"statfs":{"total":96636764160,"available":96263196672,"internally_reserved":0,"allocated":193261568,"data_stored":193067195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":180285138},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,1,0,0,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545024,"kb_used_data":296816,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93826816,"statfs":{"total":96636764160,"available":96078659584,"internally_reserved":0,"allocated":303939584,"data_stored":303745211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304107520,"data_stored":304107520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193880064,"data_stored":193880064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301469696,"data_stored":301469696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304758784,"data_stored":304758784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:27.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:27.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4692", 'reported_seq': 4742, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.903068+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:24.903068+0000', 'last_peered': '2024-09-10T20:22:24.903068+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:24.903068+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4692, 'log_dups_size': 0, 'ondisk_log_size': 4692, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38436864, 'num_objects': 4692, 'num_object_clones': 0, 'num_object_copies': 18768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4692, 'num_write_kb': 37536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2726}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:27.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:27.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4592", 'reported_seq': 4676, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:22.155913+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:22.155913+0000', 'last_peered': '2024-09-10T20:22:22.155913+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:22.155913+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4592, 'log_dups_size': 0, 'ondisk_log_size': 4592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37591632, 'num_objects': 4589, 'num_object_clones': 0, 'num_object_copies': 18356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4591, 'num_write_kb': 36718, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2627}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:27.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:27.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4711", 'reported_seq': 6987, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.609424+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:24.609424+0000', 'last_peered': '2024-09-10T20:22:24.609424+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:24.609424+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4711, 'log_dups_size': 0, 'ondisk_log_size': 4711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38556832, 'num_objects': 4707, 'num_object_clones': 0, 'num_object_copies': 18828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 903, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4709, 'num_write_kb': 37661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1104, 'num_bytes_recovered': 9011200, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3804}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:27.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:27.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4641", 'reported_seq': 4700, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.884228+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:24.884228+0000', 'last_peered': '2024-09-10T20:22:24.884228+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:24.884228+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4641, 'log_dups_size': 0, 'ondisk_log_size': 4641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37991584, 'num_objects': 4638, 'num_object_clones': 0, 'num_object_copies': 18552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4639, 'num_write_kb': 37109, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2658}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:27.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4668", 'reported_seq': 4718, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.979685+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:24.979685+0000', 'last_peered': '2024-09-10T20:22:24.979685+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:24.979685+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4668, 'log_dups_size': 0, 'ondisk_log_size': 4668, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38240256, 'num_objects': 4668, 'num_object_clones': 0, 'num_object_copies': 18672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4668, 'num_write_kb': 37344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2730}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4636", 'reported_seq': 4709, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:23.992624+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:23.992624+0000', 'last_peered': '2024-09-10T20:22:23.992624+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:23.992624+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4636, 'log_dups_size': 0, 'ondisk_log_size': 4636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37926048, 'num_objects': 4630, 'num_object_clones': 0, 'num_object_copies': 18520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4634, 'num_write_kb': 37052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2677}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4682", 'reported_seq': 4753, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.591474+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:24.591474+0000', 'last_peered': '2024-09-10T20:22:24.591474+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:24.591474+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4682, 'log_dups_size': 0, 'ondisk_log_size': 4682, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38311072, 'num_objects': 4677, 'num_object_clones': 0, 'num_object_copies': 18708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4680, 'num_write_kb': 37428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2634}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4572", 'reported_seq': 4657, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:22.116662+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:22.116662+0000', 'last_peered': '2024-09-10T20:22:22.116662+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:22.116662+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4572, 'log_dups_size': 0, 'ondisk_log_size': 4572, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37434528, 'num_objects': 4570, 'num_object_clones': 0, 'num_object_copies': 18280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4570, 'num_write_kb': 36558, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2667}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4618", 'reported_seq': 4681, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.984726+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:24.984726+0000', 'last_peered': '2024-09-10T20:22:24.984726+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:24.984726+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4618, 'log_dups_size': 0, 'ondisk_log_size': 4618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37785328, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4615, 'num_write_kb': 36908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2657}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:27.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4630", 'reported_seq': 4682, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.989451+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:24.989451+0000', 'last_peered': '2024-09-10T20:22:24.989451+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:24.989451+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4630, 'log_dups_size': 0, 'ondisk_log_size': 4630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37919312, 'num_objects': 4629, 'num_object_clones': 0, 'num_object_copies': 18516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4629, 'num_write_kb': 37031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4474", 'reported_seq': 4557, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:22.162784+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:22.162784+0000', 'last_peered': '2024-09-10T20:22:22.162784+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:22.162784+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4474, 'log_dups_size': 0, 'ondisk_log_size': 4474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36641360, 'num_objects': 4473, 'num_object_clones': 0, 'num_object_copies': 17892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4473, 'num_write_kb': 35783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4646", 'reported_seq': 4705, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.908022+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:24.908022+0000', 'last_peered': '2024-09-10T20:22:24.908022+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:24.908022+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4646, 'log_dups_size': 0, 'ondisk_log_size': 4646, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38024352, 'num_objects': 4642, 'num_object_clones': 0, 'num_object_copies': 18568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4644, 'num_write_kb': 37148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4751", 'reported_seq': 4838, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:22.157979+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:22.157979+0000', 'last_peered': '2024-09-10T20:22:22.157979+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:22.157979+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4751, 'log_dups_size': 0, 'ondisk_log_size': 4751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38892704, 'num_objects': 4748, 'num_object_clones': 0, 'num_object_copies': 18992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4749, 'num_write_kb': 37989, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:27.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4619", 'reported_seq': 4691, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:24.569475+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:24.569475+0000', 'last_peered': '2024-09-10T20:22:24.569475+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:24.569475+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4619, 'log_dups_size': 0, 'ondisk_log_size': 4619, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37794976, 'num_objects': 4614, 'num_object_clones': 0, 'num_object_copies': 18456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4617, 'num_write_kb': 36924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2635}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:27.478 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:27.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 166 20 75279 75259 3.54149 2.39844 0.0431892 0.0441078 2024-09-10T20:22:28.202 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:28.202 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26466/296508 objects degraded (8.926%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":236,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74127,"data_bytes":607214896,"bytes_used":2045530112,"bytes_avail":384501526528,"bytes_total":386547056640,"degraded_objects":26466,"degraded_total":296508,"degraded_ratio":0.089258974462746374,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":245000,"recovering_keys_per_sec":0,"num_objects_recovered":361,"num_bytes_recovered":2940928,"num_keys_recovered":0,"write_bytes_sec":2875173,"read_op_per_sec":0,"write_op_per_sec":350},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:28.238 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:28.238 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:28.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 167 20 75581 75561 3.53441 2.35938 0.0602415 0.0441942 2024-09-10T20:22:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:28.825 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:28.845 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-10T20:22:27.931194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":610630960,"num_objects":74544,"num_object_clones":0,"num_object_copies":298176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67575,"num_write_kb":540509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74603,"ondisk_log_size":74603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2007816,"kb_used_data":1083332,"kb_used_omap":14,"kb_used_meta":924401,"kb_avail":375479544,"statfs":{"total":386547056640,"available":384491053056,"internally_reserved":0,"allocated":1109331968,"data_stored":1108554476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":946587190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[19,2,8,9,14,13,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28901376,"num_objects":3528,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003791"},"pg_stats":[{"pgid":"1.f","version":"36'4692","reported_seq":4742,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.903068+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:24.903068+0000","last_peered":"2024-09-10T20:22:24.903068+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:24.903068+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38436864,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4692,"num_write_kb":37536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4645","reported_seq":4767,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:23.027797+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:23.027797+0000","last_peered":"2024-09-10T20:22:23.027797+0000","last_clean":"2024-09-10T20:22:23.027797+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:23.027797+0000","last_undegraded":"2024-09-10T20:22:23.027797+0000","last_fullsized":"2024-09-10T20:22:23.027797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38002688,"num_objects":4639,"num_object_clones":0,"num_object_copies":18556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1086,"num_write_kb":8688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4702","reported_seq":4782,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.381988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:27.381988+0000","last_peered":"2024-09-10T20:22:27.381988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:27.381988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38492752,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4701,"num_write_kb":37598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4711","reported_seq":6987,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.609424+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:24.609424+0000","last_peered":"2024-09-10T20:22:24.609424+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:24.609424+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38556832,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":37661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4641","reported_seq":4700,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.884228+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:24.884228+0000","last_peered":"2024-09-10T20:22:24.884228+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:24.884228+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37991584,"num_objects":4638,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":37109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4668","reported_seq":4718,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.979685+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:24.979685+0000","last_peered":"2024-09-10T20:22:24.979685+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:24.979685+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38240256,"num_objects":4668,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4668,"num_write_kb":37344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4636","reported_seq":4709,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:23.992624+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:23.992624+0000","last_peered":"2024-09-10T20:22:23.992624+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:23.992624+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926048,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4634,"num_write_kb":37052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2677}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4682","reported_seq":4753,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.591474+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:24.591474+0000","last_peered":"2024-09-10T20:22:24.591474+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:24.591474+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311072,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4680,"num_write_kb":37428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4609","reported_seq":8571,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:24.973873+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:24.973873+0000","last_peered":"2024-09-10T20:22:24.973873+0000","last_clean":"2024-09-10T20:22:24.973873+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:24.973873+0000","last_undegraded":"2024-09-10T20:22:24.973873+0000","last_fullsized":"2024-09-10T20:22:24.973873+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37665360,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1162,"num_write_kb":9296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4678","reported_seq":4761,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.275081+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:27.275081+0000","last_peered":"2024-09-10T20:22:27.275081+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:27.275081+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302880,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":37406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4618","reported_seq":4681,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.984726+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:24.984726+0000","last_peered":"2024-09-10T20:22:24.984726+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:24.984726+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37785328,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4630","reported_seq":4682,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.989451+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:24.989451+0000","last_peered":"2024-09-10T20:22:24.989451+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:24.989451+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37919312,"num_objects":4629,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":37031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4585","reported_seq":4668,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.360371+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:27.360371+0000","last_peered":"2024-09-10T20:22:27.360371+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:27.360371+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37550672,"num_objects":4584,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":36671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4646","reported_seq":4705,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.908022+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:24.908022+0000","last_peered":"2024-09-10T20:22:24.908022+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:24.908022+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4646,"log_dups_size":0,"ondisk_log_size":4646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38024352,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4644,"num_write_kb":37148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2667}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4841","reported_seq":4928,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.237551+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:27.237551+0000","last_peered":"2024-09-10T20:22:27.237551+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:27.237551+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39629984,"num_objects":4838,"num_object_clones":0,"num_object_copies":19352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":38709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4619","reported_seq":4691,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:24.569475+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:24.569475+0000","last_peered":"2024-09-10T20:22:24.569475+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:24.569475+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37794976,"num_objects":4614,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":36924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":610630960,"num_objects":74544,"num_object_clones":0,"num_object_copies":298176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67575,"num_write_kb":540509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":9011200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110581248,"data_stored":1110581248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74603,"ondisk_log_size":74603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545328,"kb_used_data":297056,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93826512,"statfs":{"total":96636764160,"available":96078348288,"internally_reserved":0,"allocated":304185344,"data_stored":303990971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552652,"kb_used_data":300728,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93819188,"statfs":{"total":96636764160,"available":96070848512,"internally_reserved":0,"allocated":307945472,"data_stored":307751099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,1,0,5,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364812,"kb_used_data":188732,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94007028,"statfs":{"total":96636764160,"available":96263196672,"internally_reserved":0,"allocated":193261568,"data_stored":193067195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":180285138},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,1,0,0,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545024,"kb_used_data":296816,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93826816,"statfs":{"total":96636764160,"available":96078659584,"internally_reserved":0,"allocated":303939584,"data_stored":303745211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304107520,"data_stored":304107520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193880064,"data_stored":193880064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307834880,"data_stored":307834880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304758784,"data_stored":304758784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:29.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 168 20 75929 75909 3.52955 2.71875 0.05485 0.0442537 2024-09-10T20:22:30.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 169 20 76280 76260 3.52489 2.74219 0.0576807 0.0443147 2024-09-10T20:22:31.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 170 20 76629 76609 3.52019 2.72656 0.0612766 0.0443728 2024-09-10T20:22:31.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:32.471 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:32.471 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:32.494 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-10T20:22:31.932735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":620494128,"num_objects":75748,"num_object_clones":0,"num_object_copies":302992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68779,"num_write_kb":550141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75807,"ondisk_log_size":75807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2057700,"kb_used_data":1102688,"kb_used_omap":14,"kb_used_meta":954929,"kb_avail":375429660,"statfs":{"total":386547056640,"available":384439971840,"internally_reserved":0,"allocated":1129152512,"data_stored":1128375020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":977847862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[18,1,12,6,15,16,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31449088,"num_objects":3839,"num_object_clones":0,"num_object_copies":15356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3839,"num_write_kb":30712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004029"},"pg_stats":[{"pgid":"1.f","version":"36'4789","reported_seq":4838,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.117020+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:30.117020+0000","last_peered":"2024-09-10T20:22:30.117020+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:30.117020+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39231488,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4741","reported_seq":4863,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:27.938261+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:27.938261+0000","last_peered":"2024-09-10T20:22:27.938261+0000","last_clean":"2024-09-10T20:22:27.938261+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:27.938261+0000","last_undegraded":"2024-09-10T20:22:27.938261+0000","last_fullsized":"2024-09-10T20:22:27.938261+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38789120,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1182,"num_write_kb":9456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4702","reported_seq":4782,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.381988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:27.381988+0000","last_peered":"2024-09-10T20:22:27.381988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:27.381988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38492752,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4701,"num_write_kb":37598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4803","reported_seq":7209,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.660268+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:29.660268+0000","last_peered":"2024-09-10T20:22:29.660268+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:29.660268+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310496,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4801,"num_write_kb":38397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1169,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4748","reported_seq":4808,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.030966+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:30.030966+0000","last_peered":"2024-09-10T20:22:30.030966+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:30.030966+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38868128,"num_objects":4745,"num_object_clones":0,"num_object_copies":18980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4746,"num_write_kb":37965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4784","reported_seq":4834,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.057094+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:30.057094+0000","last_peered":"2024-09-10T20:22:30.057094+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:30.057094+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":38272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4729","reported_seq":4801,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.642693+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:29.642693+0000","last_peered":"2024-09-10T20:22:29.642693+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:29.642693+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38687904,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2770}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4779","reported_seq":4850,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.574214+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:29.574214+0000","last_peered":"2024-09-10T20:22:29.574214+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:29.574214+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105696,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4712","reported_seq":8674,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:30.059684+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:30.059684+0000","last_peered":"2024-09-10T20:22:30.059684+0000","last_clean":"2024-09-10T20:22:30.059684+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:30.059684+0000","last_undegraded":"2024-09-10T20:22:30.059684+0000","last_fullsized":"2024-09-10T20:22:30.059684+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38509136,"num_objects":4701,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1265,"num_write_kb":10120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4678","reported_seq":4761,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.275081+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:27.275081+0000","last_peered":"2024-09-10T20:22:27.275081+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:27.275081+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302880,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":37406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4714","reported_seq":4777,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.074426+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:30.074426+0000","last_peered":"2024-09-10T20:22:30.074426+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:30.074426+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38571760,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4711,"num_write_kb":37676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4742","reported_seq":4792,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.063153+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:30.063153+0000","last_peered":"2024-09-10T20:22:30.063153+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:30.063153+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38836816,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4741,"num_write_kb":37927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4585","reported_seq":4668,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.360371+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:27.360371+0000","last_peered":"2024-09-10T20:22:27.360371+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:27.360371+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37550672,"num_objects":4584,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":36671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4746","reported_seq":4805,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.979717+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:29.979717+0000","last_peered":"2024-09-10T20:22:29.979717+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:29.979717+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4746,"log_dups_size":0,"ondisk_log_size":4746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38843552,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4744,"num_write_kb":37948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2767}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4841","reported_seq":4928,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.237551+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:27.237551+0000","last_peered":"2024-09-10T20:22:27.237551+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:27.237551+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39629984,"num_objects":4838,"num_object_clones":0,"num_object_copies":19352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":38709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4714","reported_seq":4786,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.703376+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:29.703376+0000","last_peered":"2024-09-10T20:22:29.703376+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:29.703376+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38573216,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":620494128,"num_objects":75748,"num_object_clones":0,"num_object_copies":302992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68779,"num_write_kb":550141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1131094016,"data_stored":1131094016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75807,"ondisk_log_size":75807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556016,"kb_used_data":303584,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93815824,"statfs":{"total":96636764160,"available":96067403776,"internally_reserved":0,"allocated":310870016,"data_stored":310675643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,0,2,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552652,"kb_used_data":300728,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93819188,"statfs":{"total":96636764160,"available":96070848512,"internally_reserved":0,"allocated":307945472,"data_stored":307751099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,1,0,5,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393524,"kb_used_data":195236,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93978316,"statfs":{"total":96636764160,"available":96233795584,"internally_reserved":0,"allocated":199921664,"data_stored":199727291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,3,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555508,"kb_used_data":303140,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93816332,"statfs":{"total":96636764160,"available":96067923968,"internally_reserved":0,"allocated":310415360,"data_stored":310220987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310923264,"data_stored":310923264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200826880,"data_stored":200826880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307834880,"data_stored":307834880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311508992,"data_stored":311508992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:32.496 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:32.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 171 20 76941 76921 3.51386 2.4375 0.0480299 0.0444553 2024-09-10T20:22:33.098 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:33.098 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:33.123 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-10T20:22:31.932735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":620494128,"num_objects":75748,"num_object_clones":0,"num_object_copies":302992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68779,"num_write_kb":550141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75807,"ondisk_log_size":75807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2057700,"kb_used_data":1102688,"kb_used_omap":14,"kb_used_meta":954929,"kb_avail":375429660,"statfs":{"total":386547056640,"available":384439971840,"internally_reserved":0,"allocated":1129152512,"data_stored":1128375020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":977847862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[18,1,12,6,15,16,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31449088,"num_objects":3839,"num_object_clones":0,"num_object_copies":15356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3839,"num_write_kb":30712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004029"},"pg_stats":[{"pgid":"1.f","version":"36'4789","reported_seq":4838,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.117020+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:30.117020+0000","last_peered":"2024-09-10T20:22:30.117020+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:30.117020+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39231488,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4741","reported_seq":4863,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:27.938261+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:27.938261+0000","last_peered":"2024-09-10T20:22:27.938261+0000","last_clean":"2024-09-10T20:22:27.938261+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:27.938261+0000","last_undegraded":"2024-09-10T20:22:27.938261+0000","last_fullsized":"2024-09-10T20:22:27.938261+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38789120,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1182,"num_write_kb":9456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4702","reported_seq":4782,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.381988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:27.381988+0000","last_peered":"2024-09-10T20:22:27.381988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:27.381988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38492752,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4701,"num_write_kb":37598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4803","reported_seq":7209,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.660268+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:29.660268+0000","last_peered":"2024-09-10T20:22:29.660268+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:29.660268+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310496,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4801,"num_write_kb":38397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1169,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4748","reported_seq":4808,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.030966+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:30.030966+0000","last_peered":"2024-09-10T20:22:30.030966+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:30.030966+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38868128,"num_objects":4745,"num_object_clones":0,"num_object_copies":18980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4746,"num_write_kb":37965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4784","reported_seq":4834,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.057094+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:30.057094+0000","last_peered":"2024-09-10T20:22:30.057094+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:30.057094+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":38272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4729","reported_seq":4801,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.642693+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:29.642693+0000","last_peered":"2024-09-10T20:22:29.642693+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:29.642693+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38687904,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2770}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4779","reported_seq":4850,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.574214+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:29.574214+0000","last_peered":"2024-09-10T20:22:29.574214+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:29.574214+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105696,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4712","reported_seq":8674,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:30.059684+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:30.059684+0000","last_peered":"2024-09-10T20:22:30.059684+0000","last_clean":"2024-09-10T20:22:30.059684+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:30.059684+0000","last_undegraded":"2024-09-10T20:22:30.059684+0000","last_fullsized":"2024-09-10T20:22:30.059684+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38509136,"num_objects":4701,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1265,"num_write_kb":10120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4678","reported_seq":4761,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.275081+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:27.275081+0000","last_peered":"2024-09-10T20:22:27.275081+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:27.275081+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302880,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":37406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4714","reported_seq":4777,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.074426+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:30.074426+0000","last_peered":"2024-09-10T20:22:30.074426+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:30.074426+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38571760,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4711,"num_write_kb":37676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4742","reported_seq":4792,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.063153+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:30.063153+0000","last_peered":"2024-09-10T20:22:30.063153+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:30.063153+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38836816,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4741,"num_write_kb":37927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4585","reported_seq":4668,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.360371+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:27.360371+0000","last_peered":"2024-09-10T20:22:27.360371+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:27.360371+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37550672,"num_objects":4584,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":36671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4746","reported_seq":4805,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.979717+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:29.979717+0000","last_peered":"2024-09-10T20:22:29.979717+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:29.979717+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4746,"log_dups_size":0,"ondisk_log_size":4746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38843552,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4744,"num_write_kb":37948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2767}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4841","reported_seq":4928,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:27.237551+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:27.237551+0000","last_peered":"2024-09-10T20:22:27.237551+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:27.237551+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39629984,"num_objects":4838,"num_object_clones":0,"num_object_copies":19352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":38709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4714","reported_seq":4786,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.703376+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:29.703376+0000","last_peered":"2024-09-10T20:22:29.703376+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:29.703376+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38573216,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":620494128,"num_objects":75748,"num_object_clones":0,"num_object_copies":302992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68779,"num_write_kb":550141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1131094016,"data_stored":1131094016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75807,"ondisk_log_size":75807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556016,"kb_used_data":303584,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93815824,"statfs":{"total":96636764160,"available":96067403776,"internally_reserved":0,"allocated":310870016,"data_stored":310675643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,0,2,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552652,"kb_used_data":300728,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93819188,"statfs":{"total":96636764160,"available":96070848512,"internally_reserved":0,"allocated":307945472,"data_stored":307751099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,1,0,5,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393524,"kb_used_data":195236,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93978316,"statfs":{"total":96636764160,"available":96233795584,"internally_reserved":0,"allocated":199921664,"data_stored":199727291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,3,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555508,"kb_used_data":303140,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93816332,"statfs":{"total":96636764160,"available":96067923968,"internally_reserved":0,"allocated":310415360,"data_stored":310220987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310923264,"data_stored":310923264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200826880,"data_stored":200826880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307834880,"data_stored":307834880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311508992,"data_stored":311508992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:33.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:33.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4789", 'reported_seq': 4838, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:30.117020+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:30.117020+0000', 'last_peered': '2024-09-10T20:22:30.117020+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:30.117020+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4789, 'log_dups_size': 0, 'ondisk_log_size': 4789, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39231488, 'num_objects': 4789, 'num_object_clones': 0, 'num_object_copies': 19156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4789, 'num_write_kb': 38312, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2823}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:33.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:33.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4702", 'reported_seq': 4782, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:27.381988+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:27.381988+0000', 'last_peered': '2024-09-10T20:22:27.381988+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:27.381988+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4702, 'log_dups_size': 0, 'ondisk_log_size': 4702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38492752, 'num_objects': 4699, 'num_object_clones': 0, 'num_object_copies': 18796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4701, 'num_write_kb': 37598, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2737}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4803", 'reported_seq': 7209, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:29.660268+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:29.660268+0000', 'last_peered': '2024-09-10T20:22:29.660268+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:29.660268+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4803, 'log_dups_size': 0, 'ondisk_log_size': 4803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39310496, 'num_objects': 4799, 'num_object_clones': 0, 'num_object_copies': 19196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 838, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4801, 'num_write_kb': 38397, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1169, 'num_bytes_recovered': 9543680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3961}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4748", 'reported_seq': 4808, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:30.030966+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:30.030966+0000', 'last_peered': '2024-09-10T20:22:30.030966+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:30.030966+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4748, 'log_dups_size': 0, 'ondisk_log_size': 4748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38868128, 'num_objects': 4745, 'num_object_clones': 0, 'num_object_copies': 18980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4746, 'num_write_kb': 37965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2765}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4784", 'reported_seq': 4834, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:30.057094+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:30.057094+0000', 'last_peered': '2024-09-10T20:22:30.057094+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:30.057094+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4784, 'log_dups_size': 0, 'ondisk_log_size': 4784, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39190528, 'num_objects': 4784, 'num_object_clones': 0, 'num_object_copies': 19136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4784, 'num_write_kb': 38272, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2846}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4729", 'reported_seq': 4801, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:29.642693+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:29.642693+0000', 'last_peered': '2024-09-10T20:22:29.642693+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:29.642693+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4729, 'log_dups_size': 0, 'ondisk_log_size': 4729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38687904, 'num_objects': 4723, 'num_object_clones': 0, 'num_object_copies': 18892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4727, 'num_write_kb': 37796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2770}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:33.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4779", 'reported_seq': 4850, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:29.574214+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:29.574214+0000', 'last_peered': '2024-09-10T20:22:29.574214+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:29.574214+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4779, 'log_dups_size': 0, 'ondisk_log_size': 4779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39105696, 'num_objects': 4774, 'num_object_clones': 0, 'num_object_copies': 19096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4777, 'num_write_kb': 38204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4678", 'reported_seq': 4761, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:27.275081+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:27.275081+0000', 'last_peered': '2024-09-10T20:22:27.275081+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:27.275081+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4678, 'log_dups_size': 0, 'ondisk_log_size': 4678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38302880, 'num_objects': 4676, 'num_object_clones': 0, 'num_object_copies': 18704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4676, 'num_write_kb': 37406, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2773}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4714", 'reported_seq': 4777, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:30.074426+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:30.074426+0000', 'last_peered': '2024-09-10T20:22:30.074426+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:30.074426+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4714, 'log_dups_size': 0, 'ondisk_log_size': 4714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38571760, 'num_objects': 4709, 'num_object_clones': 0, 'num_object_copies': 18836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4711, 'num_write_kb': 37676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2753}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4742", 'reported_seq': 4792, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:30.063153+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:30.063153+0000', 'last_peered': '2024-09-10T20:22:30.063153+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:30.063153+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4742, 'log_dups_size': 0, 'ondisk_log_size': 4742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38836816, 'num_objects': 4741, 'num_object_clones': 0, 'num_object_copies': 18964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4741, 'num_write_kb': 37927, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4585", 'reported_seq': 4668, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:27.360371+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:27.360371+0000', 'last_peered': '2024-09-10T20:22:27.360371+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:27.360371+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4585, 'log_dups_size': 0, 'ondisk_log_size': 4585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37550672, 'num_objects': 4584, 'num_object_clones': 0, 'num_object_copies': 18336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4584, 'num_write_kb': 36671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:33.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:33.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4746", 'reported_seq': 4805, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:29.979717+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:29.979717+0000', 'last_peered': '2024-09-10T20:22:29.979717+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:29.979717+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4746, 'log_dups_size': 0, 'ondisk_log_size': 4746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38843552, 'num_objects': 4742, 'num_object_clones': 0, 'num_object_copies': 18968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4744, 'num_write_kb': 37948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:33.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:33.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'4841", 'reported_seq': 4928, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:27.237551+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:27.237551+0000', 'last_peered': '2024-09-10T20:22:27.237551+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:27.237551+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4841, 'log_dups_size': 0, 'ondisk_log_size': 4841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39629984, 'num_objects': 4838, 'num_object_clones': 0, 'num_object_copies': 19352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4839, 'num_write_kb': 38709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:33.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:33.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4714", 'reported_seq': 4786, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:29.703376+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:29.703376+0000', 'last_peered': '2024-09-10T20:22:29.703376+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:29.703376+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4714, 'log_dups_size': 0, 'ondisk_log_size': 4714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38573216, 'num_objects': 4709, 'num_object_clones': 0, 'num_object_copies': 18836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4712, 'num_write_kb': 37684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2730}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:33.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:33.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 172 20 77244 77224 3.50719 2.36719 0.068058 0.0445385 2024-09-10T20:22:33.846 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:33.846 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26401/302992 objects degraded (8.713%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":242,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75748,"data_bytes":620494128,"bytes_used":2107084800,"bytes_avail":384439971840,"bytes_total":386547056640,"degraded_objects":26401,"degraded_total":302992,"degraded_ratio":0.087134313777261441,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":139217,"recovering_keys_per_sec":0,"num_objects_recovered":204,"num_bytes_recovered":1671168,"num_keys_recovered":0,"write_bytes_sec":2619877,"read_op_per_sec":0,"write_op_per_sec":319},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:33.873 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:33.873 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:34.481 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:34.482 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-10T20:22:33.933833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624999728,"num_objects":76298,"num_object_clones":0,"num_object_copies":305192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69329,"num_write_kb":554541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76357,"ondisk_log_size":76357,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2079692,"kb_used_data":1116364,"kb_used_omap":14,"kb_used_meta":963249,"kb_avail":375407668,"statfs":{"total":386547056640,"available":384417452032,"internally_reserved":0,"allocated":1143156736,"data_stored":1142379244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":986367542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[19,0,11,5,16,10,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30539776,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004535"},"pg_stats":[{"pgid":"1.f","version":"36'4789","reported_seq":4838,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.117020+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:30.117020+0000","last_peered":"2024-09-10T20:22:30.117020+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:30.117020+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39231488,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4864","reported_seq":4986,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:33.127790+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:33.127790+0000","last_peered":"2024-09-10T20:22:33.127790+0000","last_clean":"2024-09-10T20:22:33.127790+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:33.127790+0000","last_undegraded":"2024-09-10T20:22:33.127790+0000","last_fullsized":"2024-09-10T20:22:33.127790+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39796736,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":10440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4800","reported_seq":4884,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:32.358030+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:32.358030+0000","last_peered":"2024-09-10T20:22:32.358030+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:32.358030+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4800,"log_dups_size":0,"ondisk_log_size":4800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39295568,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":38382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4803","reported_seq":7209,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.660268+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:29.660268+0000","last_peered":"2024-09-10T20:22:29.660268+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:29.660268+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310496,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4801,"num_write_kb":38397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1169,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4748","reported_seq":4808,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.030966+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:30.030966+0000","last_peered":"2024-09-10T20:22:30.030966+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:30.030966+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38868128,"num_objects":4745,"num_object_clones":0,"num_object_copies":18980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4746,"num_write_kb":37965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4784","reported_seq":4834,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.057094+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:30.057094+0000","last_peered":"2024-09-10T20:22:30.057094+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:30.057094+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":38272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4729","reported_seq":4801,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.642693+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:29.642693+0000","last_peered":"2024-09-10T20:22:29.642693+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:29.642693+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38687904,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2770}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4779","reported_seq":4850,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.574214+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:29.574214+0000","last_peered":"2024-09-10T20:22:29.574214+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:29.574214+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105696,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4712","reported_seq":8674,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:30.059684+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:30.059684+0000","last_peered":"2024-09-10T20:22:30.059684+0000","last_clean":"2024-09-10T20:22:30.059684+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:30.059684+0000","last_undegraded":"2024-09-10T20:22:30.059684+0000","last_fullsized":"2024-09-10T20:22:30.059684+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38509136,"num_objects":4701,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1265,"num_write_kb":10120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4799","reported_seq":4884,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:32.341169+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:32.341169+0000","last_peered":"2024-09-10T20:22:32.341169+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:32.341169+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39294112,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4797,"num_write_kb":38374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4714","reported_seq":4777,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.074426+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:30.074426+0000","last_peered":"2024-09-10T20:22:30.074426+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:30.074426+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38571760,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4711,"num_write_kb":37676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4742","reported_seq":4792,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:30.063153+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:30.063153+0000","last_peered":"2024-09-10T20:22:30.063153+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:30.063153+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38836816,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4741,"num_write_kb":37927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4694","reported_seq":4777,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:32.196887+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:32.196887+0000","last_peered":"2024-09-10T20:22:32.196887+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:32.196887+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38443600,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4693,"num_write_kb":37543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2745}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4746","reported_seq":4805,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.979717+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:29.979717+0000","last_peered":"2024-09-10T20:22:29.979717+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:29.979717+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4746,"log_dups_size":0,"ondisk_log_size":4746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38843552,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4744,"num_write_kb":37948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2767}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'4940","reported_seq":5026,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:32.310945+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:32.310945+0000","last_peered":"2024-09-10T20:22:32.310945+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:32.310945+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4940,"log_dups_size":0,"ondisk_log_size":4940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40440992,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4938,"num_write_kb":39501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4714","reported_seq":4786,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:29.703376+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:29.703376+0000","last_peered":"2024-09-10T20:22:29.703376+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:29.703376+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38573216,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":624999728,"num_objects":76298,"num_object_clones":0,"num_object_copies":305192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69329,"num_write_kb":554541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9543680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1145102336,"data_stored":1145102336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76357,"ondisk_log_size":76357,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556016,"kb_used_data":303584,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93815824,"statfs":{"total":96636764160,"available":96067403776,"internally_reserved":0,"allocated":310870016,"data_stored":310675643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,0,2,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563512,"kb_used_data":307432,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93808328,"statfs":{"total":96636764160,"available":96059727872,"internally_reserved":0,"allocated":314810368,"data_stored":314615995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,1,0,5,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404656,"kb_used_data":202208,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93967184,"statfs":{"total":96636764160,"available":96222396416,"internally_reserved":0,"allocated":207060992,"data_stored":206866619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555508,"kb_used_data":303140,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93816332,"statfs":{"total":96636764160,"available":96067923968,"internally_reserved":0,"allocated":310415360,"data_stored":310220987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310923264,"data_stored":310923264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208031744,"data_stored":208031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":314638336,"data_stored":314638336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311508992,"data_stored":311508992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:34.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 173 20 77561 77541 3.50124 2.47656 0.0376316 0.0446152 2024-09-10T20:22:35.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 174 20 77906 77886 3.4966 2.69531 0.0219451 0.0446742 2024-09-10T20:22:36.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 175 20 78245 78225 3.49176 2.64844 0.047665 0.0447364 2024-09-10T20:22:37.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:37.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 176 20 78575 78555 3.48656 2.57812 0.0244472 0.0447991 2024-09-10T20:22:38.006 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:38.007 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:38.029 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-10T20:22:37.935219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637369648,"num_objects":77808,"num_object_clones":0,"num_object_copies":311232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70839,"num_write_kb":566621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77867,"ondisk_log_size":77867,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2095028,"kb_used_data":1135860,"kb_used_omap":14,"kb_used_meta":959089,"kb_avail":375392332,"statfs":{"total":386547056640,"available":384401747968,"internally_reserved":0,"allocated":1163120640,"data_stored":1162343148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":982107702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,7,11,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30154752,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004686"},"pg_stats":[{"pgid":"1.f","version":"36'4888","reported_seq":4938,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.120580+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:35.120580+0000","last_peered":"2024-09-10T20:22:35.120580+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:35.120580+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4864","reported_seq":4986,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:33.127790+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:33.127790+0000","last_peered":"2024-09-10T20:22:33.127790+0000","last_clean":"2024-09-10T20:22:33.127790+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:33.127790+0000","last_undegraded":"2024-09-10T20:22:33.127790+0000","last_fullsized":"2024-09-10T20:22:33.127790+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39796736,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":10440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4902","reported_seq":4986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.267941+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:37.267941+0000","last_peered":"2024-09-10T20:22:37.267941+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:37.267941+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4902,"log_dups_size":0,"ondisk_log_size":4902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40131152,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":39198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4913","reported_seq":7456,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.713851+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:34.713851+0000","last_peered":"2024-09-10T20:22:34.713851+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:34.713851+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1238,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4838","reported_seq":4898,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.130033+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:35.130033+0000","last_peered":"2024-09-10T20:22:35.130033+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:35.130033+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39605408,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4871","reported_seq":4921,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.064007+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:35.064007+0000","last_peered":"2024-09-10T20:22:35.064007+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:35.064007+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":38968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4839","reported_seq":4910,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.721895+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:34.721895+0000","last_peered":"2024-09-10T20:22:34.721895+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:34.721895+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39589024,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4837,"num_write_kb":38676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4880","reported_seq":4952,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.705393+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:34.705393+0000","last_peered":"2024-09-10T20:22:34.705393+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:34.705393+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39933088,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4809","reported_seq":8771,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:35.107704+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:35.107704+0000","last_peered":"2024-09-10T20:22:35.107704+0000","last_clean":"2024-09-10T20:22:35.107704+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:35.107704+0000","last_undegraded":"2024-09-10T20:22:35.107704+0000","last_fullsized":"2024-09-10T20:22:35.107704+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303760,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":10896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4889","reported_seq":4974,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.346275+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:37.346275+0000","last_peered":"2024-09-10T20:22:37.346275+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:37.346275+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40031392,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":39094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2984}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4827","reported_seq":4890,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.055314+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:35.055314+0000","last_peered":"2024-09-10T20:22:35.055314+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:35.055314+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39497456,"num_objects":4822,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":38580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4843","reported_seq":4893,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.151088+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:35.151088+0000","last_peered":"2024-09-10T20:22:35.151088+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:35.151088+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39664208,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4842,"num_write_kb":38735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4793","reported_seq":4876,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.297015+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:37.297015+0000","last_peered":"2024-09-10T20:22:37.297015+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:37.297015+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39254608,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4792,"num_write_kb":38335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2844}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4860","reported_seq":4919,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.068740+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:35.068740+0000","last_peered":"2024-09-10T20:22:35.068740+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:35.068740+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39777440,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5036","reported_seq":5123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.379826+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:37.379826+0000","last_peered":"2024-09-10T20:22:37.379826+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:37.379826+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41227424,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3071}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4815","reported_seq":4887,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.676023+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:34.676023+0000","last_peered":"2024-09-10T20:22:34.676023+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:34.676023+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39400608,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":637369648,"num_objects":77808,"num_object_clones":0,"num_object_copies":311232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70839,"num_write_kb":566621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165234176,"data_stored":1165234176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77867,"ondisk_log_size":77867,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566656,"kb_used_data":310064,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93805184,"statfs":{"total":96636764160,"available":96056508416,"internally_reserved":0,"allocated":317505536,"data_stored":317311163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,6,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557452,"kb_used_data":313852,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93814388,"statfs":{"total":96636764160,"available":96065933312,"internally_reserved":0,"allocated":321384448,"data_stored":321190075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404656,"kb_used_data":202208,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93967184,"statfs":{"total":96636764160,"available":96222396416,"internally_reserved":0,"allocated":207060992,"data_stored":206866619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566264,"kb_used_data":309736,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93805576,"statfs":{"total":96636764160,"available":96056909824,"internally_reserved":0,"allocated":317169664,"data_stored":316975291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317616128,"data_stored":317616128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208031744,"data_stored":208031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321335296,"data_stored":321335296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318251008,"data_stored":318251008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:38.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:38.527 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:38.558 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-10T20:22:37.935219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637369648,"num_objects":77808,"num_object_clones":0,"num_object_copies":311232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70839,"num_write_kb":566621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77867,"ondisk_log_size":77867,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2095028,"kb_used_data":1135860,"kb_used_omap":14,"kb_used_meta":959089,"kb_avail":375392332,"statfs":{"total":386547056640,"available":384401747968,"internally_reserved":0,"allocated":1163120640,"data_stored":1162343148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":982107702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,7,11,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30154752,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004686"},"pg_stats":[{"pgid":"1.f","version":"36'4888","reported_seq":4938,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.120580+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:35.120580+0000","last_peered":"2024-09-10T20:22:35.120580+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:35.120580+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4864","reported_seq":4986,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:33.127790+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:33.127790+0000","last_peered":"2024-09-10T20:22:33.127790+0000","last_clean":"2024-09-10T20:22:33.127790+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:33.127790+0000","last_undegraded":"2024-09-10T20:22:33.127790+0000","last_fullsized":"2024-09-10T20:22:33.127790+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39796736,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":10440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4902","reported_seq":4986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.267941+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:37.267941+0000","last_peered":"2024-09-10T20:22:37.267941+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:37.267941+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4902,"log_dups_size":0,"ondisk_log_size":4902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40131152,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":39198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4913","reported_seq":7456,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.713851+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:34.713851+0000","last_peered":"2024-09-10T20:22:34.713851+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:34.713851+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1238,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4838","reported_seq":4898,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.130033+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:35.130033+0000","last_peered":"2024-09-10T20:22:35.130033+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:35.130033+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39605408,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4871","reported_seq":4921,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.064007+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:35.064007+0000","last_peered":"2024-09-10T20:22:35.064007+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:35.064007+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":38968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4839","reported_seq":4910,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.721895+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:34.721895+0000","last_peered":"2024-09-10T20:22:34.721895+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:34.721895+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39589024,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4837,"num_write_kb":38676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4880","reported_seq":4952,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.705393+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:34.705393+0000","last_peered":"2024-09-10T20:22:34.705393+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:34.705393+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39933088,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4809","reported_seq":8771,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:35.107704+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:35.107704+0000","last_peered":"2024-09-10T20:22:35.107704+0000","last_clean":"2024-09-10T20:22:35.107704+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:35.107704+0000","last_undegraded":"2024-09-10T20:22:35.107704+0000","last_fullsized":"2024-09-10T20:22:35.107704+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303760,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":10896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4889","reported_seq":4974,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.346275+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:37.346275+0000","last_peered":"2024-09-10T20:22:37.346275+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:37.346275+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40031392,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":39094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2984}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4827","reported_seq":4890,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.055314+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:35.055314+0000","last_peered":"2024-09-10T20:22:35.055314+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:35.055314+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39497456,"num_objects":4822,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":38580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4843","reported_seq":4893,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.151088+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:35.151088+0000","last_peered":"2024-09-10T20:22:35.151088+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:35.151088+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39664208,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4842,"num_write_kb":38735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4793","reported_seq":4876,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.297015+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:37.297015+0000","last_peered":"2024-09-10T20:22:37.297015+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:37.297015+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39254608,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4792,"num_write_kb":38335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2844}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4860","reported_seq":4919,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.068740+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:35.068740+0000","last_peered":"2024-09-10T20:22:35.068740+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:35.068740+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39777440,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5036","reported_seq":5123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.379826+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:37.379826+0000","last_peered":"2024-09-10T20:22:37.379826+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:37.379826+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41227424,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3071}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4815","reported_seq":4887,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.676023+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:34.676023+0000","last_peered":"2024-09-10T20:22:34.676023+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:34.676023+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39400608,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":637369648,"num_objects":77808,"num_object_clones":0,"num_object_copies":311232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70839,"num_write_kb":566621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165234176,"data_stored":1165234176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77867,"ondisk_log_size":77867,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566656,"kb_used_data":310064,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93805184,"statfs":{"total":96636764160,"available":96056508416,"internally_reserved":0,"allocated":317505536,"data_stored":317311163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,6,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557452,"kb_used_data":313852,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93814388,"statfs":{"total":96636764160,"available":96065933312,"internally_reserved":0,"allocated":321384448,"data_stored":321190075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404656,"kb_used_data":202208,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93967184,"statfs":{"total":96636764160,"available":96222396416,"internally_reserved":0,"allocated":207060992,"data_stored":206866619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566264,"kb_used_data":309736,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93805576,"statfs":{"total":96636764160,"available":96056909824,"internally_reserved":0,"allocated":317169664,"data_stored":316975291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317616128,"data_stored":317616128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208031744,"data_stored":208031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321335296,"data_stored":321335296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318251008,"data_stored":318251008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4888", 'reported_seq': 4938, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:35.120580+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:35.120580+0000', 'last_peered': '2024-09-10T20:22:35.120580+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:35.120580+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4888, 'log_dups_size': 0, 'ondisk_log_size': 4888, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40042496, 'num_objects': 4888, 'num_object_clones': 0, 'num_object_copies': 19552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4888, 'num_write_kb': 39104, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2922}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'4902", 'reported_seq': 4986, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:37.267941+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:37.267941+0000', 'last_peered': '2024-09-10T20:22:37.267941+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:37.267941+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4902, 'log_dups_size': 0, 'ondisk_log_size': 4902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40131152, 'num_objects': 4899, 'num_object_clones': 0, 'num_object_copies': 19596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4901, 'num_write_kb': 39198, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'4913", 'reported_seq': 7456, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:34.713851+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:34.713851+0000', 'last_peered': '2024-09-10T20:22:34.713851+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:34.713851+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4913, 'log_dups_size': 0, 'ondisk_log_size': 4913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40211616, 'num_objects': 4909, 'num_object_clones': 0, 'num_object_copies': 19636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 769, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4911, 'num_write_kb': 39277, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1238, 'num_bytes_recovered': 10108928, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:38.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4838", 'reported_seq': 4898, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:35.130033+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:35.130033+0000', 'last_peered': '2024-09-10T20:22:35.130033+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:35.130033+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4838, 'log_dups_size': 0, 'ondisk_log_size': 4838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39605408, 'num_objects': 4835, 'num_object_clones': 0, 'num_object_copies': 19340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4836, 'num_write_kb': 38685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4871", 'reported_seq': 4921, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:35.064007+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:35.064007+0000', 'last_peered': '2024-09-10T20:22:35.064007+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:35.064007+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4871, 'log_dups_size': 0, 'ondisk_log_size': 4871, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39903232, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4871, 'num_write_kb': 38968, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2933}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4839", 'reported_seq': 4910, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:34.721895+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:34.721895+0000', 'last_peered': '2024-09-10T20:22:34.721895+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:34.721895+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4839, 'log_dups_size': 0, 'ondisk_log_size': 4839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39589024, 'num_objects': 4833, 'num_object_clones': 0, 'num_object_copies': 19332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4837, 'num_write_kb': 38676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2880}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4880", 'reported_seq': 4952, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:34.705393+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:34.705393+0000', 'last_peered': '2024-09-10T20:22:34.705393+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:34.705393+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4880, 'log_dups_size': 0, 'ondisk_log_size': 4880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39933088, 'num_objects': 4875, 'num_object_clones': 0, 'num_object_copies': 19500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4878, 'num_write_kb': 39012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4889", 'reported_seq': 4974, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:37.346275+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:37.346275+0000', 'last_peered': '2024-09-10T20:22:37.346275+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:37.346275+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4889, 'log_dups_size': 0, 'ondisk_log_size': 4889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40031392, 'num_objects': 4887, 'num_object_clones': 0, 'num_object_copies': 19548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4887, 'num_write_kb': 39094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2984}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:38.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4827", 'reported_seq': 4890, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:35.055314+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:35.055314+0000', 'last_peered': '2024-09-10T20:22:35.055314+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:35.055314+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4827, 'log_dups_size': 0, 'ondisk_log_size': 4827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39497456, 'num_objects': 4822, 'num_object_clones': 0, 'num_object_copies': 19288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4824, 'num_write_kb': 38580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2866}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4843", 'reported_seq': 4893, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:35.151088+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:35.151088+0000', 'last_peered': '2024-09-10T20:22:35.151088+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:35.151088+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4843, 'log_dups_size': 0, 'ondisk_log_size': 4843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39664208, 'num_objects': 4842, 'num_object_clones': 0, 'num_object_copies': 19368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4842, 'num_write_kb': 38735, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2864}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4793", 'reported_seq': 4876, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:37.297015+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:37.297015+0000', 'last_peered': '2024-09-10T20:22:37.297015+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:37.297015+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4793, 'log_dups_size': 0, 'ondisk_log_size': 4793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39254608, 'num_objects': 4792, 'num_object_clones': 0, 'num_object_copies': 19168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4792, 'num_write_kb': 38335, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4860", 'reported_seq': 4919, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:35.068740+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:35.068740+0000', 'last_peered': '2024-09-10T20:22:35.068740+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:35.068740+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4860, 'log_dups_size': 0, 'ondisk_log_size': 4860, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39777440, 'num_objects': 4856, 'num_object_clones': 0, 'num_object_copies': 19424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4858, 'num_write_kb': 38860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'5036", 'reported_seq': 5123, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:37.379826+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:37.379826+0000', 'last_peered': '2024-09-10T20:22:37.379826+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:37.379826+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5036, 'log_dups_size': 0, 'ondisk_log_size': 5036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41227424, 'num_objects': 5033, 'num_object_clones': 0, 'num_object_copies': 20132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5034, 'num_write_kb': 40269, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3071}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:38.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4815", 'reported_seq': 4887, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:34.676023+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:34.676023+0000', 'last_peered': '2024-09-10T20:22:34.676023+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:34.676023+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4815, 'log_dups_size': 0, 'ondisk_log_size': 4815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39400608, 'num_objects': 4810, 'num_object_clones': 0, 'num_object_copies': 19240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4813, 'num_write_kb': 38492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2831}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:38.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:38.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 177 20 78798 78778 3.4767 1.74219 0.100355 0.0449261 2024-09-10T20:22:39.276 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:39.276 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26332/311232 objects degraded (8.461%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":248,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77808,"data_bytes":637369648,"bytes_used":2145308672,"bytes_avail":384401747968,"bytes_total":386547056640,"degraded_objects":26332,"degraded_total":311232,"degraded_ratio":0.084605696072383302,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":91441,"recovering_keys_per_sec":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"write_bytes_sec":2511915,"read_op_per_sec":0,"write_op_per_sec":306},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-10T20:21:27.914153+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:39.314 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:39.314 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:39.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 178 20 79101 79081 3.47047 2.36719 0.0516818 0.0450095 2024-09-10T20:22:39.913 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:39.914 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:39.943 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-10T20:22:37.935219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637369648,"num_objects":77808,"num_object_clones":0,"num_object_copies":311232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70839,"num_write_kb":566621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77867,"ondisk_log_size":77867,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2095028,"kb_used_data":1135860,"kb_used_omap":14,"kb_used_meta":959089,"kb_avail":375392332,"statfs":{"total":386547056640,"available":384401747968,"internally_reserved":0,"allocated":1163120640,"data_stored":1162343148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":982107702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,7,11,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30154752,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004686"},"pg_stats":[{"pgid":"1.f","version":"36'4888","reported_seq":4938,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.120580+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:35.120580+0000","last_peered":"2024-09-10T20:22:35.120580+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:35.120580+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4864","reported_seq":4986,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:33.127790+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:33.127790+0000","last_peered":"2024-09-10T20:22:33.127790+0000","last_clean":"2024-09-10T20:22:33.127790+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:33.127790+0000","last_undegraded":"2024-09-10T20:22:33.127790+0000","last_fullsized":"2024-09-10T20:22:33.127790+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39796736,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":10440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4902","reported_seq":4986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.267941+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:37.267941+0000","last_peered":"2024-09-10T20:22:37.267941+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:37.267941+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4902,"log_dups_size":0,"ondisk_log_size":4902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40131152,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":39198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'4913","reported_seq":7456,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.713851+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:34.713851+0000","last_peered":"2024-09-10T20:22:34.713851+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:34.713851+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1238,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4838","reported_seq":4898,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.130033+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:35.130033+0000","last_peered":"2024-09-10T20:22:35.130033+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:35.130033+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39605408,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4871","reported_seq":4921,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.064007+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:35.064007+0000","last_peered":"2024-09-10T20:22:35.064007+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:35.064007+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":38968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4839","reported_seq":4910,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.721895+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:34.721895+0000","last_peered":"2024-09-10T20:22:34.721895+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:34.721895+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39589024,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4837,"num_write_kb":38676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4880","reported_seq":4952,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.705393+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:34.705393+0000","last_peered":"2024-09-10T20:22:34.705393+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:34.705393+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39933088,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4809","reported_seq":8771,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:35.107704+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:35.107704+0000","last_peered":"2024-09-10T20:22:35.107704+0000","last_clean":"2024-09-10T20:22:35.107704+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:35.107704+0000","last_undegraded":"2024-09-10T20:22:35.107704+0000","last_fullsized":"2024-09-10T20:22:35.107704+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303760,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":10896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4889","reported_seq":4974,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.346275+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:37.346275+0000","last_peered":"2024-09-10T20:22:37.346275+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:37.346275+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40031392,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":39094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2984}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4827","reported_seq":4890,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.055314+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:35.055314+0000","last_peered":"2024-09-10T20:22:35.055314+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:35.055314+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39497456,"num_objects":4822,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":38580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4843","reported_seq":4893,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.151088+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:35.151088+0000","last_peered":"2024-09-10T20:22:35.151088+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:35.151088+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39664208,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4842,"num_write_kb":38735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4793","reported_seq":4876,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.297015+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:37.297015+0000","last_peered":"2024-09-10T20:22:37.297015+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:37.297015+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39254608,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4792,"num_write_kb":38335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2844}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4860","reported_seq":4919,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:35.068740+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:35.068740+0000","last_peered":"2024-09-10T20:22:35.068740+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:35.068740+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39777440,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5036","reported_seq":5123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.379826+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:37.379826+0000","last_peered":"2024-09-10T20:22:37.379826+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:37.379826+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41227424,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3071}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4815","reported_seq":4887,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:34.676023+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:34.676023+0000","last_peered":"2024-09-10T20:22:34.676023+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:34.676023+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39400608,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":637369648,"num_objects":77808,"num_object_clones":0,"num_object_copies":311232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70839,"num_write_kb":566621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":10108928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165234176,"data_stored":1165234176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77867,"ondisk_log_size":77867,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566656,"kb_used_data":310064,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93805184,"statfs":{"total":96636764160,"available":96056508416,"internally_reserved":0,"allocated":317505536,"data_stored":317311163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,6,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557452,"kb_used_data":313852,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93814388,"statfs":{"total":96636764160,"available":96065933312,"internally_reserved":0,"allocated":321384448,"data_stored":321190075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404656,"kb_used_data":202208,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93967184,"statfs":{"total":96636764160,"available":96222396416,"internally_reserved":0,"allocated":207060992,"data_stored":206866619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566264,"kb_used_data":309736,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93805576,"statfs":{"total":96636764160,"available":96056909824,"internally_reserved":0,"allocated":317169664,"data_stored":316975291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317616128,"data_stored":317616128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208031744,"data_stored":208031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321335296,"data_stored":321335296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318251008,"data_stored":318251008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:40.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 179 20 79437 79417 3.46575 2.625 0.068054 0.0450736 2024-09-10T20:22:41.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:22:41.789152+0000 min lat: 0.00230373 max lat: 12.2408 avg lat: 0.0451238 2024-09-10T20:22:41.792 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:22:41.792 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 180 20 79736 79716 3.45947 2.33594 0.0318338 0.0451238 2024-09-10T20:22:42.792 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 181 20 80022 80002 3.4527 2.23438 0.0490822 0.0452397 2024-09-10T20:22:42.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:43.487 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:43.487 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:43.520 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-10T20:22:41.936535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646749488,"num_objects":78953,"num_object_clones":0,"num_object_copies":315812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71984,"num_write_kb":575781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79012,"ondisk_log_size":79012,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2093444,"kb_used_data":1155076,"kb_used_omap":14,"kb_used_meta":938289,"kb_avail":375393916,"statfs":{"total":386547056640,"available":384403369984,"internally_reserved":0,"allocated":1182797824,"data_stored":1182020332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":960808502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,9,11,16,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32243712,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004484"},"pg_stats":[{"pgid":"1.f","version":"36'4974","reported_seq":5024,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.089408+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:40.089408+0000","last_peered":"2024-09-10T20:22:40.089408+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:40.089408+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40747008,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":39792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'4976","reported_seq":5098,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:38.238215+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:38.238215+0000","last_peered":"2024-09-10T20:22:38.238215+0000","last_clean":"2024-09-10T20:22:38.238215+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:38.238215+0000","last_undegraded":"2024-09-10T20:22:38.238215+0000","last_fullsized":"2024-09-10T20:22:38.238215+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40714240,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1417,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'4902","reported_seq":4986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.267941+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:37.267941+0000","last_peered":"2024-09-10T20:22:37.267941+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:37.267941+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4902,"log_dups_size":0,"ondisk_log_size":4902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40131152,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":39198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5015","reported_seq":7743,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.692722+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:39.692722+0000","last_peered":"2024-09-10T20:22:39.692722+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:39.692722+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047200,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":40093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1330,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4939","reported_seq":4999,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.002792+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:40.002792+0000","last_peered":"2024-09-10T20:22:40.002792+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:40.002792+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40432800,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":39493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4943","reported_seq":4993,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.075315+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:40.075315+0000","last_peered":"2024-09-10T20:22:40.075315+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:40.075315+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40493056,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4943,"num_write_kb":39544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4924","reported_seq":4996,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.630639+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:39.630639+0000","last_peered":"2024-09-10T20:22:39.630639+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:39.630639+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285344,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":39356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2965}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4984","reported_seq":5056,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.762576+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:39.762576+0000","last_peered":"2024-09-10T20:22:39.762576+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:39.762576+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":39844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4905","reported_seq":8867,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:39.946992+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:39.946992+0000","last_peered":"2024-09-10T20:22:39.946992+0000","last_clean":"2024-09-10T20:22:39.946992+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:39.946992+0000","last_undegraded":"2024-09-10T20:22:39.946992+0000","last_fullsized":"2024-09-10T20:22:39.946992+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4905,"log_dups_size":0,"ondisk_log_size":4905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40090192,"num_objects":4894,"num_object_clones":0,"num_object_copies":19576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":11664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4889","reported_seq":4974,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.346275+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:37.346275+0000","last_peered":"2024-09-10T20:22:37.346275+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:37.346275+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40031392,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":39094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2984}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4933","reported_seq":4996,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.040280+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:40.040280+0000","last_peered":"2024-09-10T20:22:40.040280+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:40.040280+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40365808,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4930,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4939","reported_seq":4991,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.059225+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:40.059225+0000","last_peered":"2024-09-10T20:22:40.059225+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:40.059225+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40450640,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4938,"num_write_kb":39503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4793","reported_seq":4876,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.297015+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:37.297015+0000","last_peered":"2024-09-10T20:22:37.297015+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:37.297015+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39254608,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4792,"num_write_kb":38335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2844}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4946","reported_seq":5005,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.100849+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:40.100849+0000","last_peered":"2024-09-10T20:22:40.100849+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:40.100849+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481952,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4944,"num_write_kb":39548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5036","reported_seq":5123,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:37.379826+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:37.379826+0000","last_peered":"2024-09-10T20:22:37.379826+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:37.379826+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41227424,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3071}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4914","reported_seq":4986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.657736+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:39.657736+0000","last_peered":"2024-09-10T20:22:39.657736+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:39.657736+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":646749488,"num_objects":78953,"num_object_clones":0,"num_object_copies":315812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71984,"num_write_kb":575781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1184636928,"data_stored":1184636928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79012,"ondisk_log_size":79012,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560404,"kb_used_data":316292,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93811436,"statfs":{"total":96636764160,"available":96062910464,"internally_reserved":0,"allocated":323883008,"data_stored":323688635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557452,"kb_used_data":313852,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93814388,"statfs":{"total":96636764160,"available":96065933312,"internally_reserved":0,"allocated":321384448,"data_stored":321190075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399064,"kb_used_data":209096,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93972776,"statfs":{"total":96636764160,"available":96228122624,"internally_reserved":0,"allocated":214114304,"data_stored":213919931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":194506450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576524,"kb_used_data":315836,"kb_used_omap":4,"kb_used_meta":260667,"kb_avail":93795316,"statfs":{"total":96636764160,"available":96046403584,"internally_reserved":0,"allocated":323416064,"data_stored":323221691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":266923727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323960832,"data_stored":323960832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":214827008,"data_stored":214827008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321335296,"data_stored":321335296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324513792,"data_stored":324513792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:43.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:43.792 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 182 20 80324 80304 3.44669 2.35938 0.0496649 0.0453167 2024-09-10T20:22:44.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:44.080 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:44.118 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-10T20:22:43.937103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650583344,"num_objects":79421,"num_object_clones":0,"num_object_copies":317684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72452,"num_write_kb":579525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79480,"ondisk_log_size":79480,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2114104,"kb_used_data":1167416,"kb_used_omap":14,"kb_used_meta":946609,"kb_avail":375373256,"statfs":{"total":386547056640,"available":384382214144,"internally_reserved":0,"allocated":1195433984,"data_stored":1194656492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":969328182},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,6,7,19,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30089216,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3673,"num_write_kb":29384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004368"},"pg_stats":[{"pgid":"1.f","version":"36'4974","reported_seq":5024,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.089408+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:40.089408+0000","last_peered":"2024-09-10T20:22:40.089408+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:40.089408+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40747008,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":39792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5073","reported_seq":5195,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:43.258390+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:43.258390+0000","last_peered":"2024-09-10T20:22:43.258390+0000","last_clean":"2024-09-10T20:22:43.258390+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:43.258390+0000","last_undegraded":"2024-09-10T20:22:43.258390+0000","last_fullsized":"2024-09-10T20:22:43.258390+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41508864,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":12112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5012","reported_seq":5096,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.287740+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:42.287740+0000","last_peered":"2024-09-10T20:22:42.287740+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:42.287740+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41032272,"num_objects":5009,"num_object_clones":0,"num_object_copies":20036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":40078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3047}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5015","reported_seq":7743,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.692722+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:39.692722+0000","last_peered":"2024-09-10T20:22:39.692722+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:39.692722+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047200,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":40093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1330,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4939","reported_seq":4999,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.002792+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:40.002792+0000","last_peered":"2024-09-10T20:22:40.002792+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:40.002792+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40432800,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":39493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4943","reported_seq":4993,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.075315+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:40.075315+0000","last_peered":"2024-09-10T20:22:40.075315+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:40.075315+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40493056,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4943,"num_write_kb":39544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4924","reported_seq":4996,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.630639+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:39.630639+0000","last_peered":"2024-09-10T20:22:39.630639+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:39.630639+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285344,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":39356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2965}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4984","reported_seq":5056,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.762576+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:39.762576+0000","last_peered":"2024-09-10T20:22:39.762576+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:39.762576+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":39844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4905","reported_seq":8867,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:39.946992+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:39.946992+0000","last_peered":"2024-09-10T20:22:39.946992+0000","last_clean":"2024-09-10T20:22:39.946992+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:39.946992+0000","last_undegraded":"2024-09-10T20:22:39.946992+0000","last_fullsized":"2024-09-10T20:22:39.946992+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4905,"log_dups_size":0,"ondisk_log_size":4905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40090192,"num_objects":4894,"num_object_clones":0,"num_object_copies":19576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":11664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4972","reported_seq":5057,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.309218+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:42.309218+0000","last_peered":"2024-09-10T20:22:42.309218+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:42.309218+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40711328,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4933","reported_seq":4996,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.040280+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:40.040280+0000","last_peered":"2024-09-10T20:22:40.040280+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:40.040280+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40365808,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4930,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4939","reported_seq":4991,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.059225+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:40.059225+0000","last_peered":"2024-09-10T20:22:40.059225+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:40.059225+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40450640,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4938,"num_write_kb":39503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4881","reported_seq":4960,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.415130+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:42.415130+0000","last_peered":"2024-09-10T20:22:42.415130+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:42.415130+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39975504,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4880,"num_write_kb":39039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4946","reported_seq":5005,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.100849+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:40.100849+0000","last_peered":"2024-09-10T20:22:40.100849+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:40.100849+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481952,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4944,"num_write_kb":39548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5126","reported_seq":5211,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.329376+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:42.329376+0000","last_peered":"2024-09-10T20:22:42.329376+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:42.329376+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41964704,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4914","reported_seq":4986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.657736+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:39.657736+0000","last_peered":"2024-09-10T20:22:39.657736+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:39.657736+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":650583344,"num_objects":79421,"num_object_clones":0,"num_object_copies":317684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72452,"num_write_kb":579525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197522944,"data_stored":1197522944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79480,"ondisk_log_size":79480,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560404,"kb_used_data":316292,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93811436,"statfs":{"total":96636764160,"available":96062910464,"internally_reserved":0,"allocated":323883008,"data_stored":323688635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567452,"kb_used_data":319692,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93804388,"statfs":{"total":96636764160,"available":96055693312,"internally_reserved":0,"allocated":327364608,"data_stored":327170235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409724,"kb_used_data":215596,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93962116,"statfs":{"total":96636764160,"available":96217206784,"internally_reserved":0,"allocated":220770304,"data_stored":220575931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198766290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576524,"kb_used_data":315836,"kb_used_omap":4,"kb_used_meta":260667,"kb_avail":93795316,"statfs":{"total":96636764160,"available":96046403584,"internally_reserved":0,"allocated":323416064,"data_stored":323221691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":266923727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323960832,"data_stored":323960832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221638656,"data_stored":221638656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327409664,"data_stored":327409664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324513792,"data_stored":324513792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:44.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:44.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'4974", 'reported_seq': 5024, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:40.089408+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:40.089408+0000', 'last_peered': '2024-09-10T20:22:40.089408+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:40.089408+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4974, 'log_dups_size': 0, 'ondisk_log_size': 4974, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40747008, 'num_objects': 4974, 'num_object_clones': 0, 'num_object_copies': 19896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4974, 'num_write_kb': 39792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3008}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:44.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'5012", 'reported_seq': 5096, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:42.287740+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:42.287740+0000', 'last_peered': '2024-09-10T20:22:42.287740+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:42.287740+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5012, 'log_dups_size': 0, 'ondisk_log_size': 5012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41032272, 'num_objects': 5009, 'num_object_clones': 0, 'num_object_copies': 20036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5011, 'num_write_kb': 40078, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3047}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'5015", 'reported_seq': 7743, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:39.692722+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:39.692722+0000', 'last_peered': '2024-09-10T20:22:39.692722+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:39.692722+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5015, 'log_dups_size': 0, 'ondisk_log_size': 5015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41047200, 'num_objects': 5011, 'num_object_clones': 0, 'num_object_copies': 20044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 677, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5013, 'num_write_kb': 40093, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1330, 'num_bytes_recovered': 10862592, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'4939", 'reported_seq': 4999, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:40.002792+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:40.002792+0000', 'last_peered': '2024-09-10T20:22:40.002792+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:40.002792+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4939, 'log_dups_size': 0, 'ondisk_log_size': 4939, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40432800, 'num_objects': 4936, 'num_object_clones': 0, 'num_object_copies': 19744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4937, 'num_write_kb': 39493, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2956}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'4943", 'reported_seq': 4993, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:40.075315+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:40.075315+0000', 'last_peered': '2024-09-10T20:22:40.075315+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:40.075315+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4943, 'log_dups_size': 0, 'ondisk_log_size': 4943, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40493056, 'num_objects': 4943, 'num_object_clones': 0, 'num_object_copies': 19772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4943, 'num_write_kb': 39544, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3005}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'4924", 'reported_seq': 4996, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:39.630639+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:39.630639+0000', 'last_peered': '2024-09-10T20:22:39.630639+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:39.630639+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4924, 'log_dups_size': 0, 'ondisk_log_size': 4924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40285344, 'num_objects': 4918, 'num_object_clones': 0, 'num_object_copies': 19672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4922, 'num_write_kb': 39356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2965}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:44.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'4984", 'reported_seq': 5056, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:39.762576+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:39.762576+0000', 'last_peered': '2024-09-10T20:22:39.762576+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:39.762576+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4984, 'log_dups_size': 0, 'ondisk_log_size': 4984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40785056, 'num_objects': 4979, 'num_object_clones': 0, 'num_object_copies': 19916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4982, 'num_write_kb': 39844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'4972", 'reported_seq': 5057, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:42.309218+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:42.309218+0000', 'last_peered': '2024-09-10T20:22:42.309218+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:42.309218+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4972, 'log_dups_size': 0, 'ondisk_log_size': 4972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40711328, 'num_objects': 4970, 'num_object_clones': 0, 'num_object_copies': 19880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4970, 'num_write_kb': 39758, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3067}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'4933", 'reported_seq': 4996, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:40.040280+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:40.040280+0000', 'last_peered': '2024-09-10T20:22:40.040280+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:40.040280+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4933, 'log_dups_size': 0, 'ondisk_log_size': 4933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40365808, 'num_objects': 4928, 'num_object_clones': 0, 'num_object_copies': 19712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4930, 'num_write_kb': 39428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2972}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'4939", 'reported_seq': 4991, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:40.059225+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:40.059225+0000', 'last_peered': '2024-09-10T20:22:40.059225+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:40.059225+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4939, 'log_dups_size': 0, 'ondisk_log_size': 4939, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40450640, 'num_objects': 4938, 'num_object_clones': 0, 'num_object_copies': 19752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4938, 'num_write_kb': 39503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4881", 'reported_seq': 4960, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:42.415130+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:42.415130+0000', 'last_peered': '2024-09-10T20:22:42.415130+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:42.415130+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4881, 'log_dups_size': 0, 'ondisk_log_size': 4881, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39975504, 'num_objects': 4880, 'num_object_clones': 0, 'num_object_copies': 19520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4880, 'num_write_kb': 39039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2932}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'4946", 'reported_seq': 5005, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:40.100849+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:40.100849+0000', 'last_peered': '2024-09-10T20:22:40.100849+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:40.100849+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4946, 'log_dups_size': 0, 'ondisk_log_size': 4946, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40481952, 'num_objects': 4942, 'num_object_clones': 0, 'num_object_copies': 19768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4944, 'num_write_kb': 39548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:44.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:44.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'5126", 'reported_seq': 5211, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:42.329376+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:42.329376+0000', 'last_peered': '2024-09-10T20:22:42.329376+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:42.329376+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5126, 'log_dups_size': 0, 'ondisk_log_size': 5126, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41964704, 'num_objects': 5123, 'num_object_clones': 0, 'num_object_copies': 20492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5124, 'num_write_kb': 40989, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:44.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:44.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'4914", 'reported_seq': 4986, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:39.657736+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:39.657736+0000', 'last_peered': '2024-09-10T20:22:39.657736+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:39.657736+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4914, 'log_dups_size': 0, 'ondisk_log_size': 4914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40211616, 'num_objects': 4909, 'num_object_clones': 0, 'num_object_copies': 19636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4912, 'num_write_kb': 39284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:44.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:44.792 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 183 20 80617 80597 3.44036 2.28906 0.0295076 0.0454038 2024-09-10T20:22:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:44.869 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26240/317684 objects degraded (8.260%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":253,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79421,"data_bytes":650583344,"bytes_used":2164842496,"bytes_avail":384382214144,"bytes_total":386547056640,"degraded_objects":26240,"degraded_total":317684,"degraded_ratio":0.082597801589000389,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":109869,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"write_bytes_sec":2506522,"read_op_per_sec":0,"write_op_per_sec":305},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:44.896 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:44.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:45.416 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:45.417 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:45.429 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-10T20:22:43.937103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650583344,"num_objects":79421,"num_object_clones":0,"num_object_copies":317684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72452,"num_write_kb":579525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79480,"ondisk_log_size":79480,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2114104,"kb_used_data":1167416,"kb_used_omap":14,"kb_used_meta":946609,"kb_avail":375373256,"statfs":{"total":386547056640,"available":384382214144,"internally_reserved":0,"allocated":1195433984,"data_stored":1194656492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":969328182},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,6,7,19,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30089216,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3673,"num_write_kb":29384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004368"},"pg_stats":[{"pgid":"1.f","version":"36'4974","reported_seq":5024,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.089408+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:40.089408+0000","last_peered":"2024-09-10T20:22:40.089408+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:40.089408+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40747008,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":39792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5073","reported_seq":5195,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:43.258390+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:43.258390+0000","last_peered":"2024-09-10T20:22:43.258390+0000","last_clean":"2024-09-10T20:22:43.258390+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:43.258390+0000","last_undegraded":"2024-09-10T20:22:43.258390+0000","last_fullsized":"2024-09-10T20:22:43.258390+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41508864,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":12112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5012","reported_seq":5096,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.287740+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:42.287740+0000","last_peered":"2024-09-10T20:22:42.287740+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:42.287740+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41032272,"num_objects":5009,"num_object_clones":0,"num_object_copies":20036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":40078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3047}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5015","reported_seq":7743,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.692722+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:39.692722+0000","last_peered":"2024-09-10T20:22:39.692722+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:39.692722+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047200,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":40093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1330,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'4939","reported_seq":4999,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.002792+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:40.002792+0000","last_peered":"2024-09-10T20:22:40.002792+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:40.002792+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40432800,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":39493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'4943","reported_seq":4993,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.075315+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:40.075315+0000","last_peered":"2024-09-10T20:22:40.075315+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:40.075315+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40493056,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4943,"num_write_kb":39544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'4924","reported_seq":4996,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.630639+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:39.630639+0000","last_peered":"2024-09-10T20:22:39.630639+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:39.630639+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285344,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":39356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2965}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4984","reported_seq":5056,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.762576+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:39.762576+0000","last_peered":"2024-09-10T20:22:39.762576+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:39.762576+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":39844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4905","reported_seq":8867,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:39.946992+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:39.946992+0000","last_peered":"2024-09-10T20:22:39.946992+0000","last_clean":"2024-09-10T20:22:39.946992+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:39.946992+0000","last_undegraded":"2024-09-10T20:22:39.946992+0000","last_fullsized":"2024-09-10T20:22:39.946992+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":4905,"log_dups_size":0,"ondisk_log_size":4905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40090192,"num_objects":4894,"num_object_clones":0,"num_object_copies":19576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":11664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'4972","reported_seq":5057,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.309218+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:42.309218+0000","last_peered":"2024-09-10T20:22:42.309218+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:42.309218+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40711328,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'4933","reported_seq":4996,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.040280+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:40.040280+0000","last_peered":"2024-09-10T20:22:40.040280+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:40.040280+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40365808,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4930,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'4939","reported_seq":4991,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.059225+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:40.059225+0000","last_peered":"2024-09-10T20:22:40.059225+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:40.059225+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40450640,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4938,"num_write_kb":39503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4881","reported_seq":4960,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.415130+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:42.415130+0000","last_peered":"2024-09-10T20:22:42.415130+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:42.415130+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39975504,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4880,"num_write_kb":39039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'4946","reported_seq":5005,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:40.100849+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:40.100849+0000","last_peered":"2024-09-10T20:22:40.100849+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:40.100849+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481952,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4944,"num_write_kb":39548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5126","reported_seq":5211,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:42.329376+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:42.329376+0000","last_peered":"2024-09-10T20:22:42.329376+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:42.329376+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41964704,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'4914","reported_seq":4986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:39.657736+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:39.657736+0000","last_peered":"2024-09-10T20:22:39.657736+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:39.657736+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":650583344,"num_objects":79421,"num_object_clones":0,"num_object_copies":317684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72452,"num_write_kb":579525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197522944,"data_stored":1197522944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79480,"ondisk_log_size":79480,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560404,"kb_used_data":316292,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93811436,"statfs":{"total":96636764160,"available":96062910464,"internally_reserved":0,"allocated":323883008,"data_stored":323688635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567452,"kb_used_data":319692,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93804388,"statfs":{"total":96636764160,"available":96055693312,"internally_reserved":0,"allocated":327364608,"data_stored":327170235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409724,"kb_used_data":215596,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93962116,"statfs":{"total":96636764160,"available":96217206784,"internally_reserved":0,"allocated":220770304,"data_stored":220575931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198766290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576524,"kb_used_data":315836,"kb_used_omap":4,"kb_used_meta":260667,"kb_avail":93795316,"statfs":{"total":96636764160,"available":96046403584,"internally_reserved":0,"allocated":323416064,"data_stored":323221691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":266923727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323960832,"data_stored":323960832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221638656,"data_stored":221638656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327409664,"data_stored":327409664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324513792,"data_stored":324513792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:45.792 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 184 20 80877 80857 3.43271 2.03125 0.0491442 0.0455026 2024-09-10T20:22:46.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 185 19 81164 81145 3.42626 2.25 0.0512381 0.0455914 2024-09-10T20:22:47.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 186 20 81503 81483 3.42203 2.64062 0.111657 0.0456455 2024-09-10T20:22:48.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:48.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 187 20 81827 81807 3.41727 2.53125 0.0537641 0.0457077 2024-09-10T20:22:48.999 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:48.999 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-10T20:22:47.939923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662265136,"num_objects":80847,"num_object_clones":0,"num_object_copies":323388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73878,"num_write_kb":590933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80906,"ondisk_log_size":80906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2127956,"kb_used_data":1185428,"kb_used_omap":14,"kb_used_meta":942449,"kb_avail":375359404,"statfs":{"total":386547056640,"available":384368029696,"internally_reserved":0,"allocated":1213878272,"data_stored":1213100780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":965068342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,8,24,19,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28065792,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3426,"num_write_kb":27408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005377"},"pg_stats":[{"pgid":"1.f","version":"36'5045","reported_seq":5095,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.140956+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:45.140956+0000","last_peered":"2024-09-10T20:22:45.140956+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:45.140956+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41328640,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5045,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5073","reported_seq":5195,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:43.258390+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:43.258390+0000","last_peered":"2024-09-10T20:22:43.258390+0000","last_clean":"2024-09-10T20:22:43.258390+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:43.258390+0000","last_undegraded":"2024-09-10T20:22:43.258390+0000","last_fullsized":"2024-09-10T20:22:43.258390+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41508864,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":12112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5111","reported_seq":5194,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.390976+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:47.390976+0000","last_peered":"2024-09-10T20:22:47.390976+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:47.390976+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5111,"log_dups_size":0,"ondisk_log_size":5111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41843280,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":40870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5120","reported_seq":8128,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.721901+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:44.721901+0000","last_peered":"2024-09-10T20:22:44.721901+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:44.721901+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41907360,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5118,"num_write_kb":40933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1470,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4579}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5043","reported_seq":5103,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.010723+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:45.010723+0000","last_peered":"2024-09-10T20:22:45.010723+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:45.010723+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284768,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5029","reported_seq":5078,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.099172+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:45.099172+0000","last_peered":"2024-09-10T20:22:45.099172+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:45.099172+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5029,"log_dups_size":0,"ondisk_log_size":5029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41197568,"num_objects":5029,"num_object_clones":0,"num_object_copies":20116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":40232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5027","reported_seq":5100,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.731399+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:44.731399+0000","last_peered":"2024-09-10T20:22:44.731399+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:44.731399+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41129120,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5025,"num_write_kb":40180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5096","reported_seq":5168,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.688454+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:44.688454+0000","last_peered":"2024-09-10T20:22:44.688454+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:44.688454+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41702560,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":40740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5007","reported_seq":8969,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:45.061836+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:45.061836+0000","last_peered":"2024-09-10T20:22:45.061836+0000","last_clean":"2024-09-10T20:22:45.061836+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:45.061836+0000","last_undegraded":"2024-09-10T20:22:45.061836+0000","last_fullsized":"2024-09-10T20:22:45.061836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40925776,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5073","reported_seq":5157,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.389927+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:47.389927+0000","last_peered":"2024-09-10T20:22:47.389927+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:47.389927+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538720,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":40566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5017","reported_seq":5080,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.097712+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:45.097712+0000","last_peered":"2024-09-10T20:22:45.097712+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:45.097712+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5017,"log_dups_size":0,"ondisk_log_size":5017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41053936,"num_objects":5012,"num_object_clones":0,"num_object_copies":20048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":40100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5022","reported_seq":5074,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.123877+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:45.123877+0000","last_peered":"2024-09-10T20:22:45.123877+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:45.123877+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41130576,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3043}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4957","reported_seq":5040,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.309525+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:47.309525+0000","last_peered":"2024-09-10T20:22:47.309525+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:47.309525+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40598096,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3008}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5043","reported_seq":5102,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.081810+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:45.081810+0000","last_peered":"2024-09-10T20:22:45.081810+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:45.081810+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41276576,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5224","reported_seq":5309,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.348385+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:47.348385+0000","last_peered":"2024-09-10T20:22:47.348385+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:47.348385+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42767520,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":41773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5019","reported_seq":5090,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.741577+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:44.741577+0000","last_peered":"2024-09-10T20:22:44.741577+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:44.741577+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5019,"log_dups_size":0,"ondisk_log_size":5019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41071776,"num_objects":5014,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5017,"num_write_kb":40124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":662265136,"num_objects":80847,"num_object_clones":0,"num_object_copies":323388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73878,"num_write_kb":590933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215799296,"data_stored":1215799296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80906,"ondisk_log_size":80906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570728,"kb_used_data":322456,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93801112,"statfs":{"total":96636764160,"available":96052338688,"internally_reserved":0,"allocated":330194944,"data_stored":330000571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577332,"kb_used_data":325412,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93794508,"statfs":{"total":96636764160,"available":96045576192,"internally_reserved":0,"allocated":333221888,"data_stored":333027515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409724,"kb_used_data":215596,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93962116,"statfs":{"total":96636764160,"available":96217206784,"internally_reserved":0,"allocated":220770304,"data_stored":220575931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198766290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570172,"kb_used_data":321964,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93801668,"statfs":{"total":96636764160,"available":96052908032,"internally_reserved":0,"allocated":329691136,"data_stored":329496763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":330190848,"data_stored":330190848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221638656,"data_stored":221638656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333316096,"data_stored":333316096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330653696,"data_stored":330653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:49.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:49.650 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:49.683 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-10T20:22:47.939923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662265136,"num_objects":80847,"num_object_clones":0,"num_object_copies":323388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73878,"num_write_kb":590933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80906,"ondisk_log_size":80906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2127956,"kb_used_data":1185428,"kb_used_omap":14,"kb_used_meta":942449,"kb_avail":375359404,"statfs":{"total":386547056640,"available":384368029696,"internally_reserved":0,"allocated":1213878272,"data_stored":1213100780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":965068342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,8,24,19,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28065792,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3426,"num_write_kb":27408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005377"},"pg_stats":[{"pgid":"1.f","version":"36'5045","reported_seq":5095,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.140956+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:45.140956+0000","last_peered":"2024-09-10T20:22:45.140956+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:45.140956+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41328640,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5045,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5073","reported_seq":5195,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:43.258390+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:43.258390+0000","last_peered":"2024-09-10T20:22:43.258390+0000","last_clean":"2024-09-10T20:22:43.258390+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:43.258390+0000","last_undegraded":"2024-09-10T20:22:43.258390+0000","last_fullsized":"2024-09-10T20:22:43.258390+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41508864,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":12112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5111","reported_seq":5194,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.390976+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:47.390976+0000","last_peered":"2024-09-10T20:22:47.390976+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:47.390976+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5111,"log_dups_size":0,"ondisk_log_size":5111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41843280,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":40870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5120","reported_seq":8128,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.721901+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:44.721901+0000","last_peered":"2024-09-10T20:22:44.721901+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:44.721901+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41907360,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5118,"num_write_kb":40933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1470,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4579}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5043","reported_seq":5103,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.010723+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:45.010723+0000","last_peered":"2024-09-10T20:22:45.010723+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:45.010723+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284768,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5029","reported_seq":5078,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.099172+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:45.099172+0000","last_peered":"2024-09-10T20:22:45.099172+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:45.099172+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5029,"log_dups_size":0,"ondisk_log_size":5029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41197568,"num_objects":5029,"num_object_clones":0,"num_object_copies":20116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":40232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5027","reported_seq":5100,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.731399+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:44.731399+0000","last_peered":"2024-09-10T20:22:44.731399+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:44.731399+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41129120,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5025,"num_write_kb":40180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5096","reported_seq":5168,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.688454+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:44.688454+0000","last_peered":"2024-09-10T20:22:44.688454+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:44.688454+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41702560,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":40740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5007","reported_seq":8969,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:45.061836+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:45.061836+0000","last_peered":"2024-09-10T20:22:45.061836+0000","last_clean":"2024-09-10T20:22:45.061836+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:45.061836+0000","last_undegraded":"2024-09-10T20:22:45.061836+0000","last_fullsized":"2024-09-10T20:22:45.061836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40925776,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5073","reported_seq":5157,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.389927+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:47.389927+0000","last_peered":"2024-09-10T20:22:47.389927+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:47.389927+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538720,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":40566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5017","reported_seq":5080,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.097712+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:45.097712+0000","last_peered":"2024-09-10T20:22:45.097712+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:45.097712+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5017,"log_dups_size":0,"ondisk_log_size":5017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41053936,"num_objects":5012,"num_object_clones":0,"num_object_copies":20048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":40100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5022","reported_seq":5074,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.123877+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:45.123877+0000","last_peered":"2024-09-10T20:22:45.123877+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:45.123877+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41130576,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3043}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4957","reported_seq":5040,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.309525+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:47.309525+0000","last_peered":"2024-09-10T20:22:47.309525+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:47.309525+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40598096,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3008}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5043","reported_seq":5102,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.081810+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:45.081810+0000","last_peered":"2024-09-10T20:22:45.081810+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:45.081810+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41276576,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5224","reported_seq":5309,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.348385+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:47.348385+0000","last_peered":"2024-09-10T20:22:47.348385+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:47.348385+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42767520,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":41773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5019","reported_seq":5090,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:44.741577+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:44.741577+0000","last_peered":"2024-09-10T20:22:44.741577+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:44.741577+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5019,"log_dups_size":0,"ondisk_log_size":5019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41071776,"num_objects":5014,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5017,"num_write_kb":40124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":662265136,"num_objects":80847,"num_object_clones":0,"num_object_copies":323388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73878,"num_write_kb":590933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215799296,"data_stored":1215799296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80906,"ondisk_log_size":80906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570728,"kb_used_data":322456,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93801112,"statfs":{"total":96636764160,"available":96052338688,"internally_reserved":0,"allocated":330194944,"data_stored":330000571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577332,"kb_used_data":325412,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93794508,"statfs":{"total":96636764160,"available":96045576192,"internally_reserved":0,"allocated":333221888,"data_stored":333027515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409724,"kb_used_data":215596,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93962116,"statfs":{"total":96636764160,"available":96217206784,"internally_reserved":0,"allocated":220770304,"data_stored":220575931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198766290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570172,"kb_used_data":321964,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93801668,"statfs":{"total":96636764160,"available":96052908032,"internally_reserved":0,"allocated":329691136,"data_stored":329496763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":330190848,"data_stored":330190848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221638656,"data_stored":221638656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333316096,"data_stored":333316096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330653696,"data_stored":330653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:49.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:49.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'5045", 'reported_seq': 5095, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:45.140956+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:45.140956+0000', 'last_peered': '2024-09-10T20:22:45.140956+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:45.140956+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5045, 'log_dups_size': 0, 'ondisk_log_size': 5045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41328640, 'num_objects': 5045, 'num_object_clones': 0, 'num_object_copies': 20180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5045, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3079}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:49.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:49.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'5111", 'reported_seq': 5194, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:47.390976+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:47.390976+0000', 'last_peered': '2024-09-10T20:22:47.390976+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:47.390976+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5111, 'log_dups_size': 0, 'ondisk_log_size': 5111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41843280, 'num_objects': 5108, 'num_object_clones': 0, 'num_object_copies': 20432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5110, 'num_write_kb': 40870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3146}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:49.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:49.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'5120", 'reported_seq': 8128, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:44.721901+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:44.721901+0000', 'last_peered': '2024-09-10T20:22:44.721901+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:44.721901+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5120, 'log_dups_size': 0, 'ondisk_log_size': 5120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41907360, 'num_objects': 5116, 'num_object_clones': 0, 'num_object_copies': 20464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5118, 'num_write_kb': 40933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1470, 'num_bytes_recovered': 12009472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4579}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:49.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'5043", 'reported_seq': 5103, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:45.010723+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:45.010723+0000', 'last_peered': '2024-09-10T20:22:45.010723+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:45.010723+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5043, 'log_dups_size': 0, 'ondisk_log_size': 5043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41284768, 'num_objects': 5040, 'num_object_clones': 0, 'num_object_copies': 20160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5041, 'num_write_kb': 40325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3060}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'5029", 'reported_seq': 5078, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:45.099172+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:45.099172+0000', 'last_peered': '2024-09-10T20:22:45.099172+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:45.099172+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5029, 'log_dups_size': 0, 'ondisk_log_size': 5029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41197568, 'num_objects': 5029, 'num_object_clones': 0, 'num_object_copies': 20116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5029, 'num_write_kb': 40232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3091}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'5027", 'reported_seq': 5100, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:44.731399+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:44.731399+0000', 'last_peered': '2024-09-10T20:22:44.731399+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:44.731399+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5027, 'log_dups_size': 0, 'ondisk_log_size': 5027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41129120, 'num_objects': 5021, 'num_object_clones': 0, 'num_object_copies': 20084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5025, 'num_write_kb': 40180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3068}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'5096", 'reported_seq': 5168, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:44.688454+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:44.688454+0000', 'last_peered': '2024-09-10T20:22:44.688454+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:44.688454+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5096, 'log_dups_size': 0, 'ondisk_log_size': 5096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41702560, 'num_objects': 5091, 'num_object_clones': 0, 'num_object_copies': 20364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5094, 'num_write_kb': 40740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3048}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'5073", 'reported_seq': 5157, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:47.389927+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:47.389927+0000', 'last_peered': '2024-09-10T20:22:47.389927+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:47.389927+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5073, 'log_dups_size': 0, 'ondisk_log_size': 5073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41538720, 'num_objects': 5071, 'num_object_clones': 0, 'num_object_copies': 20284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5071, 'num_write_kb': 40566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3168}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:49.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'5017", 'reported_seq': 5080, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:45.097712+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:45.097712+0000', 'last_peered': '2024-09-10T20:22:45.097712+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:45.097712+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5017, 'log_dups_size': 0, 'ondisk_log_size': 5017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41053936, 'num_objects': 5012, 'num_object_clones': 0, 'num_object_copies': 20048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5014, 'num_write_kb': 40100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3056}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'5022", 'reported_seq': 5074, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:45.123877+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:45.123877+0000', 'last_peered': '2024-09-10T20:22:45.123877+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:45.123877+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5022, 'log_dups_size': 0, 'ondisk_log_size': 5022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41130576, 'num_objects': 5021, 'num_object_clones': 0, 'num_object_copies': 20084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5021, 'num_write_kb': 40167, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'4957", 'reported_seq': 5040, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:47.309525+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:47.309525+0000', 'last_peered': '2024-09-10T20:22:47.309525+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:47.309525+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 4957, 'log_dups_size': 0, 'ondisk_log_size': 4957, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40598096, 'num_objects': 4956, 'num_object_clones': 0, 'num_object_copies': 19824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4956, 'num_write_kb': 39647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'5043", 'reported_seq': 5102, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:45.081810+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:45.081810+0000', 'last_peered': '2024-09-10T20:22:45.081810+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:45.081810+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5043, 'log_dups_size': 0, 'ondisk_log_size': 5043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41276576, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 20156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5041, 'num_write_kb': 40324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'5224", 'reported_seq': 5309, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:47.348385+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:47.348385+0000', 'last_peered': '2024-09-10T20:22:47.348385+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:47.348385+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5224, 'log_dups_size': 0, 'ondisk_log_size': 5224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42767520, 'num_objects': 5221, 'num_object_clones': 0, 'num_object_copies': 20884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5222, 'num_write_kb': 41773, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:49.686 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'5019", 'reported_seq': 5090, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:44.741577+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:44.741577+0000', 'last_peered': '2024-09-10T20:22:44.741577+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:44.741577+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5019, 'log_dups_size': 0, 'ondisk_log_size': 5019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41071776, 'num_objects': 5014, 'num_object_clones': 0, 'num_object_copies': 20056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5017, 'num_write_kb': 40124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:49.687 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:49.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 188 20 82108 82088 3.41077 2.19531 0.0647412 0.0457956 2024-09-10T20:22:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:50.420 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25962/325412 objects degraded (7.978%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":259,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81353,"data_bytes":666410288,"bytes_used":2200358912,"bytes_avail":384346697728,"bytes_total":386547056640,"degraded_objects":25962,"degraded_total":325412,"degraded_ratio":0.079781937974014477,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":252473,"recovering_keys_per_sec":0,"num_objects_recovered":370,"num_bytes_recovered":3031040,"num_keys_recovered":0,"write_bytes_sec":2418967,"read_op_per_sec":0,"write_op_per_sec":295},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:50.441 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:50.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:50.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 189 20 82402 82382 3.40487 2.29688 0.0562062 0.0458776 2024-09-10T20:22:50.995 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:50.995 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:51.030 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-10T20:22:49.940604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666410288,"num_objects":81353,"num_object_clones":0,"num_object_copies":325412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74384,"num_write_kb":594981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1628,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81412,"ondisk_log_size":81412,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2148788,"kb_used_data":1197940,"kb_used_omap":14,"kb_used_meta":950769,"kb_avail":375338572,"statfs":{"total":386547056640,"available":384346697728,"internally_reserved":0,"allocated":1226690560,"data_stored":1225913068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":973588022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,4,8,16,18,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29040640,"num_objects":3545,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":3031040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005385"},"pg_stats":[{"pgid":"1.f","version":"36'5045","reported_seq":5095,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.140956+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:45.140956+0000","last_peered":"2024-09-10T20:22:45.140956+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:45.140956+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41328640,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5045,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5184","reported_seq":5306,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:48.356699+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:48.356699+0000","last_peered":"2024-09-10T20:22:48.356699+0000","last_clean":"2024-09-10T20:22:48.356699+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:48.356699+0000","last_undegraded":"2024-09-10T20:22:48.356699+0000","last_fullsized":"2024-09-10T20:22:48.356699+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5184,"log_dups_size":0,"ondisk_log_size":5184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42418176,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1625,"num_write_kb":13000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5111","reported_seq":5194,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.390976+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:47.390976+0000","last_peered":"2024-09-10T20:22:47.390976+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:47.390976+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5111,"log_dups_size":0,"ondisk_log_size":5111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41843280,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":40870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5220","reported_seq":8503,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.790984+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:49.790984+0000","last_peered":"2024-09-10T20:22:49.790984+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:49.790984+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42726560,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1608,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4817}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5043","reported_seq":5103,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.010723+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:45.010723+0000","last_peered":"2024-09-10T20:22:45.010723+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:45.010723+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284768,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5029","reported_seq":5078,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.099172+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:45.099172+0000","last_peered":"2024-09-10T20:22:45.099172+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:45.099172+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5029,"log_dups_size":0,"ondisk_log_size":5029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41197568,"num_objects":5029,"num_object_clones":0,"num_object_copies":20116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":40232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5119","reported_seq":5192,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.767723+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:49.767723+0000","last_peered":"2024-09-10T20:22:49.767723+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:49.767723+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41882784,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5117,"num_write_kb":40916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3160}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5194","reported_seq":5265,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.792972+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:49.792972+0000","last_peered":"2024-09-10T20:22:49.792972+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:49.792972+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42505376,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":41524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5007","reported_seq":8969,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:45.061836+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:45.061836+0000","last_peered":"2024-09-10T20:22:45.061836+0000","last_clean":"2024-09-10T20:22:45.061836+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:45.061836+0000","last_undegraded":"2024-09-10T20:22:45.061836+0000","last_fullsized":"2024-09-10T20:22:45.061836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40925776,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5073","reported_seq":5157,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.389927+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:47.389927+0000","last_peered":"2024-09-10T20:22:47.389927+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:47.389927+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538720,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":40566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5017","reported_seq":5080,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.097712+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:45.097712+0000","last_peered":"2024-09-10T20:22:45.097712+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:45.097712+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5017,"log_dups_size":0,"ondisk_log_size":5017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41053936,"num_objects":5012,"num_object_clones":0,"num_object_copies":20048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":40100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5022","reported_seq":5074,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.123877+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:45.123877+0000","last_peered":"2024-09-10T20:22:45.123877+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:45.123877+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41130576,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3043}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4957","reported_seq":5040,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.309525+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:47.309525+0000","last_peered":"2024-09-10T20:22:47.309525+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:47.309525+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40598096,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3008}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5043","reported_seq":5102,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:45.081810+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:45.081810+0000","last_peered":"2024-09-10T20:22:45.081810+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:45.081810+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41276576,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5224","reported_seq":5309,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:47.348385+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:47.348385+0000","last_peered":"2024-09-10T20:22:47.348385+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:47.348385+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42767520,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":41773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5124","reported_seq":5195,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.747574+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:49.747574+0000","last_peered":"2024-09-10T20:22:49.747574+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:49.747574+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41931936,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":666410288,"num_objects":81353,"num_object_clones":0,"num_object_copies":325412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74384,"num_write_kb":594981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1628,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1228746752,"data_stored":1228746752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81412,"ondisk_log_size":81412,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570728,"kb_used_data":322456,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93801112,"statfs":{"total":96636764160,"available":96052338688,"internally_reserved":0,"allocated":330194944,"data_stored":330000571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577332,"kb_used_data":325412,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93794508,"statfs":{"total":96636764160,"available":96045576192,"internally_reserved":0,"allocated":333221888,"data_stored":333027515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420340,"kb_used_data":222052,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93951500,"statfs":{"total":96636764160,"available":96206336000,"internally_reserved":0,"allocated":227381248,"data_stored":227186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580388,"kb_used_data":328020,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93791452,"statfs":{"total":96636764160,"available":96042446848,"internally_reserved":0,"allocated":335892480,"data_stored":335698107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,3,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336371712,"data_stored":336371712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228405248,"data_stored":228405248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333316096,"data_stored":333316096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330653696,"data_stored":330653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:51.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 190 20 82729 82709 3.4004 2.55469 0.0463908 0.0459385 2024-09-10T20:22:52.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 191 20 83059 83039 3.39609 2.57812 0.0448538 0.0459973 2024-09-10T20:22:53.795 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 192 20 83344 83324 3.39 2.22656 0.112268 0.046078 2024-09-10T20:22:54.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:54.620 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:54.642 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-10T20:22:53.942304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":675487024,"num_objects":82461,"num_object_clones":0,"num_object_copies":329844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75492,"num_write_kb":603845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1628,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82520,"ondisk_log_size":82520,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2180000,"kb_used_data":1216668,"kb_used_omap":14,"kb_used_meta":963249,"kb_avail":375307360,"statfs":{"total":386547056640,"available":384314736640,"internally_reserved":0,"allocated":1245868032,"data_stored":1245090540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":986367542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,8,6,9,14,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":2277376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005769"},"pg_stats":[{"pgid":"1.f","version":"36'5133","reported_seq":5183,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.060787+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:50.060787+0000","last_peered":"2024-09-10T20:22:50.060787+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:50.060787+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42049536,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":41064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5288","reported_seq":5410,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:53.241871+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:53.241871+0000","last_peered":"2024-09-10T20:22:53.241871+0000","last_clean":"2024-09-10T20:22:53.241871+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:53.241871+0000","last_undegraded":"2024-09-10T20:22:53.241871+0000","last_fullsized":"2024-09-10T20:22:53.241871+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43270144,"num_objects":5282,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1729,"num_write_kb":13832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5204","reported_seq":5288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.375215+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:52.375215+0000","last_peered":"2024-09-10T20:22:52.375215+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:52.375215+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605136,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":41614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5220","reported_seq":8503,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.790984+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:49.790984+0000","last_peered":"2024-09-10T20:22:49.790984+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:49.790984+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42726560,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1608,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4817}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5133","reported_seq":5193,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.724968+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:49.724968+0000","last_peered":"2024-09-10T20:22:49.724968+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:49.724968+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42022048,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5131,"num_write_kb":41045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5124","reported_seq":5174,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.160208+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:50.160208+0000","last_peered":"2024-09-10T20:22:50.160208+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:50.160208+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41975808,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5119","reported_seq":5192,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.767723+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:49.767723+0000","last_peered":"2024-09-10T20:22:49.767723+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:49.767723+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41882784,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5117,"num_write_kb":40916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3160}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5194","reported_seq":5265,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.792972+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:49.792972+0000","last_peered":"2024-09-10T20:22:49.792972+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:49.792972+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42505376,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":41524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5093","reported_seq":9055,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:50.187452+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:50.187452+0000","last_peered":"2024-09-10T20:22:50.187452+0000","last_clean":"2024-09-10T20:22:50.187452+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:50.187452+0000","last_undegraded":"2024-09-10T20:22:50.187452+0000","last_fullsized":"2024-09-10T20:22:50.187452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41630288,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":13168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5165","reported_seq":5250,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.374966+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:52.374966+0000","last_peered":"2024-09-10T20:22:52.374966+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:52.374966+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42292384,"num_objects":5163,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5163,"num_write_kb":41302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5110","reported_seq":5173,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.006061+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:50.006061+0000","last_peered":"2024-09-10T20:22:50.006061+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:50.006061+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41815792,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":40844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5102","reported_seq":5154,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.128616+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:50.128616+0000","last_peered":"2024-09-10T20:22:50.128616+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:50.128616+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41785936,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5101,"num_write_kb":40807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3123}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5066","reported_seq":5148,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.441216+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:52.441216+0000","last_peered":"2024-09-10T20:22:52.441216+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:52.441216+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41491024,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5065,"num_write_kb":40519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5129","reported_seq":5188,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.122780+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:50.122780+0000","last_peered":"2024-09-10T20:22:50.122780+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:50.122780+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41981088,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5127,"num_write_kb":41012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5316","reported_seq":5403,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.415058+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:52.415058+0000","last_peered":"2024-09-10T20:22:52.415058+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:52.415058+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43521184,"num_objects":5313,"num_object_clones":0,"num_object_copies":21252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5314,"num_write_kb":42509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5124","reported_seq":5195,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.747574+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:49.747574+0000","last_peered":"2024-09-10T20:22:49.747574+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:49.747574+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41931936,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":675487024,"num_objects":82461,"num_object_clones":0,"num_object_copies":329844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75492,"num_write_kb":603845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1628,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1248038912,"data_stored":1248038912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82520,"ondisk_log_size":82520,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580736,"kb_used_data":328300,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93791104,"statfs":{"total":96636764160,"available":96042090496,"internally_reserved":0,"allocated":336179200,"data_stored":335984827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587844,"kb_used_data":331764,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93783996,"statfs":{"total":96636764160,"available":96034811904,"internally_reserved":0,"allocated":339726336,"data_stored":339531963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431032,"kb_used_data":228584,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93940808,"statfs":{"total":96636764160,"available":96195387392,"internally_reserved":0,"allocated":234070016,"data_stored":233875643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,0,0,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580388,"kb_used_data":328020,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93791452,"statfs":{"total":96636764160,"available":96042446848,"internally_reserved":0,"allocated":335892480,"data_stored":335698107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,3,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336371712,"data_stored":336371712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235077632,"data_stored":235077632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339755008,"data_stored":339755008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336834560,"data_stored":336834560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:54.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:54.796 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 193 20 83646 83626 3.38466 2.35938 0.0538316 0.0461486 2024-09-10T20:22:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:55.219 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-10T20:22:53.942304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":675487024,"num_objects":82461,"num_object_clones":0,"num_object_copies":329844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75492,"num_write_kb":603845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1628,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82520,"ondisk_log_size":82520,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2180000,"kb_used_data":1216668,"kb_used_omap":14,"kb_used_meta":963249,"kb_avail":375307360,"statfs":{"total":386547056640,"available":384314736640,"internally_reserved":0,"allocated":1245868032,"data_stored":1245090540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":986367542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,8,6,9,14,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":2277376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005769"},"pg_stats":[{"pgid":"1.f","version":"36'5133","reported_seq":5183,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.060787+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:50.060787+0000","last_peered":"2024-09-10T20:22:50.060787+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:50.060787+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42049536,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":41064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5288","reported_seq":5410,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:53.241871+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:53.241871+0000","last_peered":"2024-09-10T20:22:53.241871+0000","last_clean":"2024-09-10T20:22:53.241871+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:53.241871+0000","last_undegraded":"2024-09-10T20:22:53.241871+0000","last_fullsized":"2024-09-10T20:22:53.241871+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43270144,"num_objects":5282,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1729,"num_write_kb":13832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5204","reported_seq":5288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.375215+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:52.375215+0000","last_peered":"2024-09-10T20:22:52.375215+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:52.375215+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605136,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":41614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5220","reported_seq":8503,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.790984+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:49.790984+0000","last_peered":"2024-09-10T20:22:49.790984+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:49.790984+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42726560,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1608,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4817}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5133","reported_seq":5193,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.724968+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:49.724968+0000","last_peered":"2024-09-10T20:22:49.724968+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:49.724968+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42022048,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5131,"num_write_kb":41045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5124","reported_seq":5174,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.160208+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:50.160208+0000","last_peered":"2024-09-10T20:22:50.160208+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:50.160208+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41975808,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5119","reported_seq":5192,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.767723+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:49.767723+0000","last_peered":"2024-09-10T20:22:49.767723+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:49.767723+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41882784,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5117,"num_write_kb":40916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3160}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5194","reported_seq":5265,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.792972+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:49.792972+0000","last_peered":"2024-09-10T20:22:49.792972+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:49.792972+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42505376,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":41524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5093","reported_seq":9055,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:50.187452+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:50.187452+0000","last_peered":"2024-09-10T20:22:50.187452+0000","last_clean":"2024-09-10T20:22:50.187452+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:50.187452+0000","last_undegraded":"2024-09-10T20:22:50.187452+0000","last_fullsized":"2024-09-10T20:22:50.187452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41630288,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":13168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5165","reported_seq":5250,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.374966+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:52.374966+0000","last_peered":"2024-09-10T20:22:52.374966+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:52.374966+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42292384,"num_objects":5163,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5163,"num_write_kb":41302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5110","reported_seq":5173,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.006061+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:50.006061+0000","last_peered":"2024-09-10T20:22:50.006061+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:50.006061+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41815792,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":40844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5102","reported_seq":5154,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.128616+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:50.128616+0000","last_peered":"2024-09-10T20:22:50.128616+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:50.128616+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41785936,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5101,"num_write_kb":40807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3123}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5066","reported_seq":5148,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.441216+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:52.441216+0000","last_peered":"2024-09-10T20:22:52.441216+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:52.441216+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41491024,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5065,"num_write_kb":40519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5129","reported_seq":5188,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:50.122780+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:50.122780+0000","last_peered":"2024-09-10T20:22:50.122780+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:50.122780+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41981088,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5127,"num_write_kb":41012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5316","reported_seq":5403,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.415058+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:52.415058+0000","last_peered":"2024-09-10T20:22:52.415058+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:52.415058+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43521184,"num_objects":5313,"num_object_clones":0,"num_object_copies":21252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5314,"num_write_kb":42509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5124","reported_seq":5195,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:49.747574+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:49.747574+0000","last_peered":"2024-09-10T20:22:49.747574+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:49.747574+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41931936,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":675487024,"num_objects":82461,"num_object_clones":0,"num_object_copies":329844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75492,"num_write_kb":603845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1628,"num_bytes_recovered":13139968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1248038912,"data_stored":1248038912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82520,"ondisk_log_size":82520,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580736,"kb_used_data":328300,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93791104,"statfs":{"total":96636764160,"available":96042090496,"internally_reserved":0,"allocated":336179200,"data_stored":335984827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587844,"kb_used_data":331764,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93783996,"statfs":{"total":96636764160,"available":96034811904,"internally_reserved":0,"allocated":339726336,"data_stored":339531963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431032,"kb_used_data":228584,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93940808,"statfs":{"total":96636764160,"available":96195387392,"internally_reserved":0,"allocated":234070016,"data_stored":233875643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,0,0,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580388,"kb_used_data":328020,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93791452,"statfs":{"total":96636764160,"available":96042446848,"internally_reserved":0,"allocated":335892480,"data_stored":335698107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,3,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336371712,"data_stored":336371712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235077632,"data_stored":235077632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339755008,"data_stored":339755008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336834560,"data_stored":336834560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'5133", 'reported_seq': 5183, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:50.060787+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:50.060787+0000', 'last_peered': '2024-09-10T20:22:50.060787+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:50.060787+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5133, 'log_dups_size': 0, 'ondisk_log_size': 5133, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42049536, 'num_objects': 5133, 'num_object_clones': 0, 'num_object_copies': 20532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5133, 'num_write_kb': 41064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'5204", 'reported_seq': 5288, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:52.375215+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:52.375215+0000', 'last_peered': '2024-09-10T20:22:52.375215+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:52.375215+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5204, 'log_dups_size': 0, 'ondisk_log_size': 5204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42605136, 'num_objects': 5201, 'num_object_clones': 0, 'num_object_copies': 20804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5203, 'num_write_kb': 41614, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3239}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'5220", 'reported_seq': 8503, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:49.790984+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:49.790984+0000', 'last_peered': '2024-09-10T20:22:49.790984+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:49.790984+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5220, 'log_dups_size': 0, 'ondisk_log_size': 5220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42726560, 'num_objects': 5216, 'num_object_clones': 0, 'num_object_copies': 20864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 399, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5218, 'num_write_kb': 41733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1608, 'num_bytes_recovered': 13139968, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4817}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:22:55.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'5133", 'reported_seq': 5193, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:49.724968+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:49.724968+0000', 'last_peered': '2024-09-10T20:22:49.724968+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:49.724968+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5133, 'log_dups_size': 0, 'ondisk_log_size': 5133, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42022048, 'num_objects': 5130, 'num_object_clones': 0, 'num_object_copies': 20520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5131, 'num_write_kb': 41045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3150}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'5124", 'reported_seq': 5174, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:50.160208+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:50.160208+0000', 'last_peered': '2024-09-10T20:22:50.160208+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:50.160208+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5124, 'log_dups_size': 0, 'ondisk_log_size': 5124, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41975808, 'num_objects': 5124, 'num_object_clones': 0, 'num_object_copies': 20496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5124, 'num_write_kb': 40992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'5119", 'reported_seq': 5192, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:49.767723+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:49.767723+0000', 'last_peered': '2024-09-10T20:22:49.767723+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:49.767723+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5119, 'log_dups_size': 0, 'ondisk_log_size': 5119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41882784, 'num_objects': 5113, 'num_object_clones': 0, 'num_object_copies': 20452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5117, 'num_write_kb': 40916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3160}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'5194", 'reported_seq': 5265, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:49.792972+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:49.792972+0000', 'last_peered': '2024-09-10T20:22:49.792972+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:49.792972+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5194, 'log_dups_size': 0, 'ondisk_log_size': 5194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42505376, 'num_objects': 5189, 'num_object_clones': 0, 'num_object_copies': 20756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5192, 'num_write_kb': 41524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3146}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:22:55.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'5165", 'reported_seq': 5250, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:52.374966+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:52.374966+0000', 'last_peered': '2024-09-10T20:22:52.374966+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:52.374966+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5165, 'log_dups_size': 0, 'ondisk_log_size': 5165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42292384, 'num_objects': 5163, 'num_object_clones': 0, 'num_object_copies': 20652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5163, 'num_write_kb': 41302, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3260}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'5110", 'reported_seq': 5173, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:50.006061+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:50.006061+0000', 'last_peered': '2024-09-10T20:22:50.006061+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:50.006061+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5110, 'log_dups_size': 0, 'ondisk_log_size': 5110, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41815792, 'num_objects': 5105, 'num_object_clones': 0, 'num_object_copies': 20420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5107, 'num_write_kb': 40844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3149}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'5102", 'reported_seq': 5154, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:50.128616+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:50.128616+0000', 'last_peered': '2024-09-10T20:22:50.128616+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:50.128616+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5102, 'log_dups_size': 0, 'ondisk_log_size': 5102, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41785936, 'num_objects': 5101, 'num_object_clones': 0, 'num_object_copies': 20404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5101, 'num_write_kb': 40807, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3123}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'5066", 'reported_seq': 5148, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:52.441216+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:52.441216+0000', 'last_peered': '2024-09-10T20:22:52.441216+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:52.441216+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5066, 'log_dups_size': 0, 'ondisk_log_size': 5066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41491024, 'num_objects': 5065, 'num_object_clones': 0, 'num_object_copies': 20260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5065, 'num_write_kb': 40519, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3117}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'5129", 'reported_seq': 5188, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:50.122780+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:50.122780+0000', 'last_peered': '2024-09-10T20:22:50.122780+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:50.122780+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5129, 'log_dups_size': 0, 'ondisk_log_size': 5129, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41981088, 'num_objects': 5125, 'num_object_clones': 0, 'num_object_copies': 20500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5127, 'num_write_kb': 41012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:22:55.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'5316", 'reported_seq': 5403, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:52.415058+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:52.415058+0000', 'last_peered': '2024-09-10T20:22:52.415058+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:52.415058+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5316, 'log_dups_size': 0, 'ondisk_log_size': 5316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43521184, 'num_objects': 5313, 'num_object_clones': 0, 'num_object_copies': 21252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5314, 'num_write_kb': 42509, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:22:55.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:22:55.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'5124", 'reported_seq': 5195, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:49.747574+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:49.747574+0000', 'last_peered': '2024-09-10T20:22:49.747574+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:49.747574+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5124, 'log_dups_size': 0, 'ondisk_log_size': 5124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41931936, 'num_objects': 5119, 'num_object_clones': 0, 'num_object_copies': 20476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5122, 'num_write_kb': 40964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:22:55.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:22:55.796 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 194 20 83941 83921 3.37908 2.30469 0.0654742 0.0462254 2024-09-10T20:22:55.973 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25962/329844 objects degraded (7.871%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":264,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82461,"data_bytes":675487024,"bytes_used":2232320000,"bytes_avail":384314736640,"bytes_total":386547056640,"degraded_objects":25962,"degraded_total":329844,"degraded_ratio":0.078709935605922807,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":189690,"recovering_keys_per_sec":0,"num_objects_recovered":278,"num_bytes_recovered":2277376,"num_keys_recovered":0,"write_bytes_sec":2393643,"read_op_per_sec":0,"write_op_per_sec":292},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:22:56.002 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:22:56.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:56.592 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:22:56.592 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:22:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-10T20:22:55.942939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":684105008,"num_objects":83513,"num_object_clones":0,"num_object_copies":334052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76544,"num_write_kb":612261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1742,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83572,"ondisk_log_size":83572,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2236512,"kb_used_data":1228768,"kb_used_omap":14,"kb_used_meta":1007665,"kb_avail":375250848,"statfs":{"total":386547056640,"available":384256868352,"internally_reserved":0,"allocated":1258258432,"data_stored":1257480940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1031849526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,5,8,6,12,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33521664,"num_objects":4092,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4092,"num_write_kb":32736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":3211264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005836"},"pg_stats":[{"pgid":"1.f","version":"36'5229","reported_seq":5279,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.141750+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:55.141750+0000","last_peered":"2024-09-10T20:22:55.141750+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:55.141750+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5229,"log_dups_size":0,"ondisk_log_size":5229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42835968,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5288","reported_seq":5410,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:53.241871+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:53.241871+0000","last_peered":"2024-09-10T20:22:53.241871+0000","last_clean":"2024-09-10T20:22:53.241871+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:53.241871+0000","last_undegraded":"2024-09-10T20:22:53.241871+0000","last_fullsized":"2024-09-10T20:22:53.241871+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43270144,"num_objects":5282,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1729,"num_write_kb":13832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5204","reported_seq":5288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.375215+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:52.375215+0000","last_peered":"2024-09-10T20:22:52.375215+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:52.375215+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605136,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":41614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5319","reported_seq":8831,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.841983+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:54.841983+0000","last_peered":"2024-09-10T20:22:54.841983+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:54.841983+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43537568,"num_objects":5315,"num_object_clones":0,"num_object_copies":21260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":42525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1722,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5227","reported_seq":5287,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.129736+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:55.129736+0000","last_peered":"2024-09-10T20:22:55.129736+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:55.129736+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42792096,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5209","reported_seq":5259,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.185260+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:55.185260+0000","last_peered":"2024-09-10T20:22:55.185260+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:55.185260+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42672128,"num_objects":5209,"num_object_clones":0,"num_object_copies":20836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":41672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5220","reported_seq":5288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.867007+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:54.867007+0000","last_peered":"2024-09-10T20:22:54.867007+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:54.867007+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5292","reported_seq":5364,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.738440+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:54.738440+0000","last_peered":"2024-09-10T20:22:54.738440+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:54.738440+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43308192,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5290,"num_write_kb":42308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5207","reported_seq":9169,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:55.124200+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:55.124200+0000","last_peered":"2024-09-10T20:22:55.124200+0000","last_clean":"2024-09-10T20:22:55.124200+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:55.124200+0000","last_undegraded":"2024-09-10T20:22:55.124200+0000","last_fullsized":"2024-09-10T20:22:55.124200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42564176,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":14080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5165","reported_seq":5250,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.374966+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:52.374966+0000","last_peered":"2024-09-10T20:22:52.374966+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:52.374966+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42292384,"num_objects":5163,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5163,"num_write_kb":41302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5206","reported_seq":5269,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.143495+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:55.143495+0000","last_peered":"2024-09-10T20:22:55.143495+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:55.143495+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42602224,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5197","reported_seq":5249,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.157087+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:55.157087+0000","last_peered":"2024-09-10T20:22:55.157087+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:55.157087+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5197,"log_dups_size":0,"ondisk_log_size":5197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42564176,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":41567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5066","reported_seq":5148,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.441216+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:52.441216+0000","last_peered":"2024-09-10T20:22:52.441216+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:52.441216+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41491024,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5065,"num_write_kb":40519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5218","reported_seq":5277,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.058767+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:55.058767+0000","last_peered":"2024-09-10T20:22:55.058767+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:55.058767+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5316","reported_seq":5403,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:52.415058+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:52.415058+0000","last_peered":"2024-09-10T20:22:52.415058+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:52.415058+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43521184,"num_objects":5313,"num_object_clones":0,"num_object_copies":21252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5314,"num_write_kb":42509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5209","reported_seq":5281,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.879820+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:54.879820+0000","last_peered":"2024-09-10T20:22:54.879820+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:54.879820+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628256,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5207,"num_write_kb":41644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":684105008,"num_objects":83513,"num_object_clones":0,"num_object_copies":334052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76544,"num_write_kb":612261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1742,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1260736512,"data_stored":1260736512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83572,"ondisk_log_size":83572,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608964,"kb_used_data":334324,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93762876,"statfs":{"total":96636764160,"available":96013185024,"internally_reserved":0,"allocated":342347776,"data_stored":342153403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":281210576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587844,"kb_used_data":331764,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93783996,"statfs":{"total":96636764160,"available":96034811904,"internally_reserved":0,"allocated":339726336,"data_stored":339531963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431032,"kb_used_data":228584,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93940808,"statfs":{"total":96636764160,"available":96195387392,"internally_reserved":0,"allocated":234070016,"data_stored":233875643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,0,0,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608672,"kb_used_data":334096,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93763168,"statfs":{"total":96636764160,"available":96013484032,"internally_reserved":0,"allocated":342114304,"data_stored":341919931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":342777856,"data_stored":342777856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235077632,"data_stored":235077632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339755008,"data_stored":339755008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343126016,"data_stored":343126016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:22:56.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 195 20 84245 84225 3.37393 2.375 0.064318 0.0462994 2024-09-10T20:22:57.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 196 20 84567 84547 3.36955 2.51562 0.158989 0.046357 2024-09-10T20:22:58.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 197 20 84896 84876 3.36549 2.57031 0.0505623 0.0464169 2024-09-10T20:22:59.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:22:59.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 198 20 85218 85198 3.3612 2.51562 0.0493738 0.0464717 2024-09-10T20:23:00.252 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:00.253 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:00.273 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-10T20:22:59.944470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":688069936,"num_objects":83997,"num_object_clones":0,"num_object_copies":335988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77028,"num_write_kb":616133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1742,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84056,"ondisk_log_size":84056,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2260128,"kb_used_data":1241248,"kb_used_omap":14,"kb_used_meta":1018801,"kb_avail":375227232,"statfs":{"total":386547056640,"available":384232685568,"internally_reserved":0,"allocated":1271037952,"data_stored":1270260460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1043252790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,13,6,30,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25804800,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":252,"num_bytes_recovered":2064384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004546"},"pg_stats":[{"pgid":"1.f","version":"36'5229","reported_seq":5279,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.141750+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:55.141750+0000","last_peered":"2024-09-10T20:22:55.141750+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:55.141750+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5229,"log_dups_size":0,"ondisk_log_size":5229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42835968,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5371","reported_seq":5493,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:58.138038+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:58.138038+0000","last_peered":"2024-09-10T20:22:58.138038+0000","last_clean":"2024-09-10T20:22:58.138038+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:58.138038+0000","last_undegraded":"2024-09-10T20:22:58.138038+0000","last_fullsized":"2024-09-10T20:22:58.138038+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43950080,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1812,"num_write_kb":14496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5313","reported_seq":5397,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.512088+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:57.512088+0000","last_peered":"2024-09-10T20:22:57.512088+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:57.512088+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43498064,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3348}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5319","reported_seq":8831,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.841983+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:54.841983+0000","last_peered":"2024-09-10T20:22:54.841983+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:54.841983+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43537568,"num_objects":5315,"num_object_clones":0,"num_object_copies":21260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":42525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1722,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5227","reported_seq":5287,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.129736+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:55.129736+0000","last_peered":"2024-09-10T20:22:55.129736+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:55.129736+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42792096,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5209","reported_seq":5259,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.185260+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:55.185260+0000","last_peered":"2024-09-10T20:22:55.185260+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:55.185260+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42672128,"num_objects":5209,"num_object_clones":0,"num_object_copies":20836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":41672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5220","reported_seq":5288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.867007+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:54.867007+0000","last_peered":"2024-09-10T20:22:54.867007+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:54.867007+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5292","reported_seq":5364,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.738440+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:54.738440+0000","last_peered":"2024-09-10T20:22:54.738440+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:54.738440+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43308192,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5290,"num_write_kb":42308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5207","reported_seq":9169,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:55.124200+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:55.124200+0000","last_peered":"2024-09-10T20:22:55.124200+0000","last_clean":"2024-09-10T20:22:55.124200+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:55.124200+0000","last_undegraded":"2024-09-10T20:22:55.124200+0000","last_fullsized":"2024-09-10T20:22:55.124200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42564176,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":14080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5265","reported_seq":5349,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.512094+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:57.512094+0000","last_peered":"2024-09-10T20:22:57.512094+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:57.512094+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43111584,"num_objects":5263,"num_object_clones":0,"num_object_copies":21052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5206","reported_seq":5269,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.143495+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:55.143495+0000","last_peered":"2024-09-10T20:22:55.143495+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:55.143495+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42602224,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5197","reported_seq":5249,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.157087+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:55.157087+0000","last_peered":"2024-09-10T20:22:55.157087+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:55.157087+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5197,"log_dups_size":0,"ondisk_log_size":5197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42564176,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":41567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5159","reported_seq":5242,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.494799+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:57.494799+0000","last_peered":"2024-09-10T20:22:57.494799+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:57.494799+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42252880,"num_objects":5158,"num_object_clones":0,"num_object_copies":20632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":41263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3210}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5218","reported_seq":5277,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.058767+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:55.058767+0000","last_peered":"2024-09-10T20:22:55.058767+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:55.058767+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5415","reported_seq":5502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.449182+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:57.449182+0000","last_peered":"2024-09-10T20:22:57.449182+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:57.449182+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5415,"log_dups_size":0,"ondisk_log_size":5415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44332192,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5413,"num_write_kb":43301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5209","reported_seq":5281,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.879820+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:54.879820+0000","last_peered":"2024-09-10T20:22:54.879820+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:54.879820+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628256,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5207,"num_write_kb":41644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":688069936,"num_objects":83997,"num_object_clones":0,"num_object_copies":335988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77028,"num_write_kb":616133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1742,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1273716736,"data_stored":1273716736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84056,"ondisk_log_size":84056,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608964,"kb_used_data":334324,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93762876,"statfs":{"total":96636764160,"available":96013185024,"internally_reserved":0,"allocated":342347776,"data_stored":342153403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":281210576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599424,"kb_used_data":337776,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93772416,"statfs":{"total":96636764160,"available":96022953984,"internally_reserved":0,"allocated":345882624,"data_stored":345688251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,15,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443068,"kb_used_data":235052,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93928772,"statfs":{"total":96636764160,"available":96183062528,"internally_reserved":0,"allocated":240693248,"data_stored":240498875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212987602},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608672,"kb_used_data":334096,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93763168,"statfs":{"total":96636764160,"available":96013484032,"internally_reserved":0,"allocated":342114304,"data_stored":341919931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":342777856,"data_stored":342777856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241803264,"data_stored":241803264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346009600,"data_stored":346009600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343126016,"data_stored":343126016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:00.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:00.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 199 20 85489 85469 3.35494 2.11719 0.125409 0.0465583 2024-09-10T20:23:00.813 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:00.813 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:00.836 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-10T20:22:59.944470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":688069936,"num_objects":83997,"num_object_clones":0,"num_object_copies":335988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77028,"num_write_kb":616133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1742,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84056,"ondisk_log_size":84056,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2260128,"kb_used_data":1241248,"kb_used_omap":14,"kb_used_meta":1018801,"kb_avail":375227232,"statfs":{"total":386547056640,"available":384232685568,"internally_reserved":0,"allocated":1271037952,"data_stored":1270260460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1043252790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,13,6,30,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25804800,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":252,"num_bytes_recovered":2064384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004546"},"pg_stats":[{"pgid":"1.f","version":"36'5229","reported_seq":5279,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.141750+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:22:55.141750+0000","last_peered":"2024-09-10T20:22:55.141750+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:22:55.141750+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5229,"log_dups_size":0,"ondisk_log_size":5229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42835968,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5371","reported_seq":5493,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:58.138038+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:58.138038+0000","last_peered":"2024-09-10T20:22:58.138038+0000","last_clean":"2024-09-10T20:22:58.138038+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:58.138038+0000","last_undegraded":"2024-09-10T20:22:58.138038+0000","last_fullsized":"2024-09-10T20:22:58.138038+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43950080,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1812,"num_write_kb":14496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5313","reported_seq":5397,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.512088+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:57.512088+0000","last_peered":"2024-09-10T20:22:57.512088+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:57.512088+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43498064,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3348}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5319","reported_seq":8831,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.841983+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:54.841983+0000","last_peered":"2024-09-10T20:22:54.841983+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:54.841983+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43537568,"num_objects":5315,"num_object_clones":0,"num_object_copies":21260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":42525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1722,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5227","reported_seq":5287,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.129736+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:22:55.129736+0000","last_peered":"2024-09-10T20:22:55.129736+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:22:55.129736+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42792096,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5209","reported_seq":5259,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.185260+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:22:55.185260+0000","last_peered":"2024-09-10T20:22:55.185260+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:22:55.185260+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42672128,"num_objects":5209,"num_object_clones":0,"num_object_copies":20836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":41672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5220","reported_seq":5288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.867007+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:54.867007+0000","last_peered":"2024-09-10T20:22:54.867007+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:54.867007+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5292","reported_seq":5364,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.738440+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:54.738440+0000","last_peered":"2024-09-10T20:22:54.738440+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:54.738440+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43308192,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5290,"num_write_kb":42308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5207","reported_seq":9169,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:55.124200+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:22:55.124200+0000","last_peered":"2024-09-10T20:22:55.124200+0000","last_clean":"2024-09-10T20:22:55.124200+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:22:55.124200+0000","last_undegraded":"2024-09-10T20:22:55.124200+0000","last_fullsized":"2024-09-10T20:22:55.124200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42564176,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":14080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5265","reported_seq":5349,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.512094+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:57.512094+0000","last_peered":"2024-09-10T20:22:57.512094+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:57.512094+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43111584,"num_objects":5263,"num_object_clones":0,"num_object_copies":21052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5206","reported_seq":5269,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.143495+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:22:55.143495+0000","last_peered":"2024-09-10T20:22:55.143495+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:22:55.143495+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42602224,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5197","reported_seq":5249,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.157087+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:22:55.157087+0000","last_peered":"2024-09-10T20:22:55.157087+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:22:55.157087+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5197,"log_dups_size":0,"ondisk_log_size":5197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42564176,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":41567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5159","reported_seq":5242,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.494799+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:57.494799+0000","last_peered":"2024-09-10T20:22:57.494799+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:57.494799+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42252880,"num_objects":5158,"num_object_clones":0,"num_object_copies":20632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":41263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3210}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5218","reported_seq":5277,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:55.058767+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:22:55.058767+0000","last_peered":"2024-09-10T20:22:55.058767+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:22:55.058767+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5415","reported_seq":5502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.449182+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:57.449182+0000","last_peered":"2024-09-10T20:22:57.449182+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:57.449182+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5415,"log_dups_size":0,"ondisk_log_size":5415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44332192,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5413,"num_write_kb":43301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5209","reported_seq":5281,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:54.879820+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:54.879820+0000","last_peered":"2024-09-10T20:22:54.879820+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:54.879820+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628256,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5207,"num_write_kb":41644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":688069936,"num_objects":83997,"num_object_clones":0,"num_object_copies":335988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77028,"num_write_kb":616133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1742,"num_bytes_recovered":14073856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1273716736,"data_stored":1273716736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84056,"ondisk_log_size":84056,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608964,"kb_used_data":334324,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93762876,"statfs":{"total":96636764160,"available":96013185024,"internally_reserved":0,"allocated":342347776,"data_stored":342153403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":281210576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599424,"kb_used_data":337776,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93772416,"statfs":{"total":96636764160,"available":96022953984,"internally_reserved":0,"allocated":345882624,"data_stored":345688251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,15,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443068,"kb_used_data":235052,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93928772,"statfs":{"total":96636764160,"available":96183062528,"internally_reserved":0,"allocated":240693248,"data_stored":240498875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212987602},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608672,"kb_used_data":334096,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93763168,"statfs":{"total":96636764160,"available":96013484032,"internally_reserved":0,"allocated":342114304,"data_stored":341919931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":342777856,"data_stored":342777856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241803264,"data_stored":241803264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346009600,"data_stored":346009600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343126016,"data_stored":343126016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:00.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:00.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'5229", 'reported_seq': 5279, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:55.141750+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:22:55.141750+0000', 'last_peered': '2024-09-10T20:22:55.141750+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:22:55.141750+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5229, 'log_dups_size': 0, 'ondisk_log_size': 5229, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42835968, 'num_objects': 5229, 'num_object_clones': 0, 'num_object_copies': 20916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5229, 'num_write_kb': 41832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:00.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'5313", 'reported_seq': 5397, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:57.512088+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:22:57.512088+0000', 'last_peered': '2024-09-10T20:22:57.512088+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:22:57.512088+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5313, 'log_dups_size': 0, 'ondisk_log_size': 5313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43498064, 'num_objects': 5310, 'num_object_clones': 0, 'num_object_copies': 21240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5312, 'num_write_kb': 42486, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3348}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'5319", 'reported_seq': 8831, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:54.841983+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:22:54.841983+0000', 'last_peered': '2024-09-10T20:22:54.841983+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:22:54.841983+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5319, 'log_dups_size': 0, 'ondisk_log_size': 5319, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43537568, 'num_objects': 5315, 'num_object_clones': 0, 'num_object_copies': 21260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5317, 'num_write_kb': 42525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1722, 'num_bytes_recovered': 14073856, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5030}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'5227", 'reported_seq': 5287, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:55.129736+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:22:55.129736+0000', 'last_peered': '2024-09-10T20:22:55.129736+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:22:55.129736+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5227, 'log_dups_size': 0, 'ondisk_log_size': 5227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42792096, 'num_objects': 5224, 'num_object_clones': 0, 'num_object_copies': 20896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5225, 'num_write_kb': 41797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3244}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'5209", 'reported_seq': 5259, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:55.185260+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:22:55.185260+0000', 'last_peered': '2024-09-10T20:22:55.185260+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:22:55.185260+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5209, 'log_dups_size': 0, 'ondisk_log_size': 5209, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42672128, 'num_objects': 5209, 'num_object_clones': 0, 'num_object_copies': 20836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5209, 'num_write_kb': 41672, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3271}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'5220", 'reported_seq': 5288, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:54.867007+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:22:54.867007+0000', 'last_peered': '2024-09-10T20:22:54.867007+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:22:54.867007+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5220, 'log_dups_size': 0, 'ondisk_log_size': 5220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42710176, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5218, 'num_write_kb': 41724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3261}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:00.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'5292", 'reported_seq': 5364, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:54.738440+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:22:54.738440+0000', 'last_peered': '2024-09-10T20:22:54.738440+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:22:54.738440+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5292, 'log_dups_size': 0, 'ondisk_log_size': 5292, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43308192, 'num_objects': 5287, 'num_object_clones': 0, 'num_object_copies': 21148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5290, 'num_write_kb': 42308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3244}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'5265", 'reported_seq': 5349, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:57.512094+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:22:57.512094+0000', 'last_peered': '2024-09-10T20:22:57.512094+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:22:57.512094+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5265, 'log_dups_size': 0, 'ondisk_log_size': 5265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43111584, 'num_objects': 5263, 'num_object_clones': 0, 'num_object_copies': 21052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5263, 'num_write_kb': 42102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3360}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'5206", 'reported_seq': 5269, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:55.143495+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:22:55.143495+0000', 'last_peered': '2024-09-10T20:22:55.143495+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:22:55.143495+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5206, 'log_dups_size': 0, 'ondisk_log_size': 5206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42602224, 'num_objects': 5201, 'num_object_clones': 0, 'num_object_copies': 20804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5203, 'num_write_kb': 41612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3245}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'5197", 'reported_seq': 5249, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:55.157087+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:22:55.157087+0000', 'last_peered': '2024-09-10T20:22:55.157087+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:22:55.157087+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5197, 'log_dups_size': 0, 'ondisk_log_size': 5197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42564176, 'num_objects': 5196, 'num_object_clones': 0, 'num_object_copies': 20784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5196, 'num_write_kb': 41567, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'5159", 'reported_seq': 5242, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:57.494799+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:22:57.494799+0000', 'last_peered': '2024-09-10T20:22:57.494799+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:22:57.494799+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5159, 'log_dups_size': 0, 'ondisk_log_size': 5159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42252880, 'num_objects': 5158, 'num_object_clones': 0, 'num_object_copies': 20632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5158, 'num_write_kb': 41263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:00.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'5218", 'reported_seq': 5277, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:55.058767+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:22:55.058767+0000', 'last_peered': '2024-09-10T20:22:55.058767+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:22:55.058767+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5218, 'log_dups_size': 0, 'ondisk_log_size': 5218, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42710176, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5216, 'num_write_kb': 41724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:00.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:00.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'5415", 'reported_seq': 5502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:57.449182+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:22:57.449182+0000', 'last_peered': '2024-09-10T20:22:57.449182+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:22:57.449182+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5415, 'log_dups_size': 0, 'ondisk_log_size': 5415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44332192, 'num_objects': 5412, 'num_object_clones': 0, 'num_object_copies': 21648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5413, 'num_write_kb': 43301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:00.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:00.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'5209", 'reported_seq': 5281, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:22:54.879820+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:22:54.879820+0000', 'last_peered': '2024-09-10T20:22:54.879820+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:22:54.879820+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5209, 'log_dups_size': 0, 'ondisk_log_size': 5209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42628256, 'num_objects': 5204, 'num_object_clones': 0, 'num_object_copies': 20816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5207, 'num_write_kb': 41644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3225}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:00.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:01.605 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:01.606 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25848/335988 objects degraded (7.693%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":270,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83997,"data_bytes":688069936,"bytes_used":2314371072,"bytes_avail":384232685568,"bytes_total":386547056640,"degraded_objects":25848,"degraded_total":335988,"degraded_ratio":0.076931318975677704,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":171966,"recovering_keys_per_sec":0,"num_objects_recovered":252,"num_bytes_recovered":2064384,"num_keys_recovered":0,"write_bytes_sec":2149585,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:01.635 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:01.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:01.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:23:01.794565+0000 min lat: 0.00230373 max lat: 12.2408 avg lat: 0.0466338 2024-09-10T20:23:01.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:23:01.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 200 20 85781 85761 3.34958 2.28125 0.0494359 0.0466338 2024-09-10T20:23:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:02.149 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:02.169 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-10T20:23:01.945144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":697138480,"num_objects":85104,"num_object_clones":0,"num_object_copies":340416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78135,"num_write_kb":624989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1820,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85163,"ondisk_log_size":85163,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2247436,"kb_used_data":1253516,"kb_used_omap":14,"kb_used_meta":993841,"kb_avail":375239924,"statfs":{"total":386547056640,"available":384245682176,"internally_reserved":0,"allocated":1283600384,"data_stored":1282822892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1017693750},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,5,8,16,29,24,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30728192,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004539"},"pg_stats":[{"pgid":"1.f","version":"36'5328","reported_seq":5377,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.126088+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:00.126088+0000","last_peered":"2024-09-10T20:23:00.126088+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:00.126088+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5328,"log_dups_size":0,"ondisk_log_size":5328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43646976,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5328,"num_write_kb":42624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5371","reported_seq":5493,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:22:58.138038+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:22:58.138038+0000","last_peered":"2024-09-10T20:22:58.138038+0000","last_clean":"2024-09-10T20:22:58.138038+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:22:58.138038+0000","last_undegraded":"2024-09-10T20:22:58.138038+0000","last_fullsized":"2024-09-10T20:22:58.138038+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43950080,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1812,"num_write_kb":14496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5313","reported_seq":5397,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.512088+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:22:57.512088+0000","last_peered":"2024-09-10T20:22:57.512088+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:22:57.512088+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43498064,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3348}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5438","reported_seq":9106,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.956821+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:59.956821+0000","last_peered":"2024-09-10T20:22:59.956821+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:59.956821+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5438,"log_dups_size":0,"ondisk_log_size":5438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44512416,"num_objects":5434,"num_object_clones":0,"num_object_copies":21736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5436,"num_write_kb":43477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5314","reported_seq":5374,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.193358+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:00.193358+0000","last_peered":"2024-09-10T20:23:00.193358+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:00.193358+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43504800,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5297","reported_seq":5347,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.106724+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:00.106724+0000","last_peered":"2024-09-10T20:23:00.106724+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:00.106724+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43393024,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5297,"num_write_kb":42376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5331","reported_seq":5404,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.723710+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:59.723710+0000","last_peered":"2024-09-10T20:22:59.723710+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:59.723710+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43619488,"num_objects":5325,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":42612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3372}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5386","reported_seq":5458,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.871811+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:59.871811+0000","last_peered":"2024-09-10T20:22:59.871811+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:59.871811+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5386,"log_dups_size":0,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44078240,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5384,"num_write_kb":43060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5311","reported_seq":9273,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:00.114574+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:00.114574+0000","last_peered":"2024-09-10T20:23:00.114574+0000","last_clean":"2024-09-10T20:23:00.114574+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:00.114574+0000","last_undegraded":"2024-09-10T20:23:00.114574+0000","last_fullsized":"2024-09-10T20:23:00.114574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43416144,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1864,"num_write_kb":14912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5265","reported_seq":5349,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.512094+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:22:57.512094+0000","last_peered":"2024-09-10T20:22:57.512094+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:22:57.512094+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43111584,"num_objects":5263,"num_object_clones":0,"num_object_copies":21052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5302","reported_seq":5363,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.074605+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:00.074605+0000","last_peered":"2024-09-10T20:23:00.074605+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:00.074605+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5302,"log_dups_size":0,"ondisk_log_size":5302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43388656,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":42380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5309","reported_seq":5359,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.230358+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:00.230358+0000","last_peered":"2024-09-10T20:23:00.230358+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:00.230358+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43481680,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5308,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5159","reported_seq":5242,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.494799+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:22:57.494799+0000","last_peered":"2024-09-10T20:22:57.494799+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:22:57.494799+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42252880,"num_objects":5158,"num_object_clones":0,"num_object_copies":20632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":41263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3210}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5314","reported_seq":5370,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.171621+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:00.171621+0000","last_peered":"2024-09-10T20:23:00.171621+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:00.171621+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43496608,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5415","reported_seq":5502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:57.449182+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:22:57.449182+0000","last_peered":"2024-09-10T20:22:57.449182+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:22:57.449182+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5415,"log_dups_size":0,"ondisk_log_size":5415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44332192,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5413,"num_write_kb":43301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5310","reported_seq":5382,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.975783+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:59.975783+0000","last_peered":"2024-09-10T20:22:59.975783+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:59.975783+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5310,"log_dups_size":0,"ondisk_log_size":5310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43455648,"num_objects":5305,"num_object_clones":0,"num_object_copies":21220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5308,"num_write_kb":42452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":697138480,"num_objects":85104,"num_object_clones":0,"num_object_copies":340416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78135,"num_write_kb":624989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1820,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1286680576,"data_stored":1286680576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85163,"ondisk_log_size":85163,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602656,"kb_used_data":340496,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93769184,"statfs":{"total":96636764160,"available":96019644416,"internally_reserved":0,"allocated":348667904,"data_stored":348473531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":268431056},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599424,"kb_used_data":337776,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93772416,"statfs":{"total":96636764160,"available":96022953984,"internally_reserved":0,"allocated":345882624,"data_stored":345688251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,15,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443068,"kb_used_data":235052,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93928772,"statfs":{"total":96636764160,"available":96183062528,"internally_reserved":0,"allocated":240693248,"data_stored":240498875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212987602},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602288,"kb_used_data":340192,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93769552,"statfs":{"total":96636764160,"available":96020021248,"internally_reserved":0,"allocated":348356608,"data_stored":348162235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,4,1,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":349270016,"data_stored":349270016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241803264,"data_stored":241803264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346009600,"data_stored":346009600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":349597696,"data_stored":349597696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:02.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 201 20 86059 86039 3.34372 2.17188 0.0411923 0.0467194 2024-09-10T20:23:03.797 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 202 20 86382 86362 3.33965 2.52344 0.0300145 0.0467714 2024-09-10T20:23:04.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 203 20 86713 86693 3.33593 2.58594 0.087351 0.0468247 2024-09-10T20:23:05.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:05.756 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:05.757 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-10T20:23:03.945708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":701013296,"num_objects":85577,"num_object_clones":0,"num_object_copies":342308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78608,"num_write_kb":628773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1820,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85636,"ondisk_log_size":85636,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2305548,"kb_used_data":1267212,"kb_used_omap":14,"kb_used_meta":1038257,"kb_avail":375181812,"statfs":{"total":386547056640,"available":384186175488,"internally_reserved":0,"allocated":1297625088,"data_stored":1296847596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1063175734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,4,5,17,12,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29540352,"num_objects":3606,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":28848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004246"},"pg_stats":[{"pgid":"1.f","version":"36'5328","reported_seq":5377,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.126088+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:00.126088+0000","last_peered":"2024-09-10T20:23:00.126088+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:00.126088+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5328,"log_dups_size":0,"ondisk_log_size":5328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43646976,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5328,"num_write_kb":42624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5465","reported_seq":5585,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:03.473141+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:03.473141+0000","last_peered":"2024-09-10T20:23:03.473141+0000","last_clean":"2024-09-10T20:23:03.473141+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:03.473141+0000","last_undegraded":"2024-09-10T20:23:03.473141+0000","last_fullsized":"2024-09-10T20:23:03.473141+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44720128,"num_objects":5459,"num_object_clones":0,"num_object_copies":21836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1906,"num_write_kb":15248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5404","reported_seq":5488,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.544267+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:02.544267+0000","last_peered":"2024-09-10T20:23:02.544267+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:02.544267+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5404,"log_dups_size":0,"ondisk_log_size":5404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243536,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":43214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5438","reported_seq":9106,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.956821+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:22:59.956821+0000","last_peered":"2024-09-10T20:22:59.956821+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:22:59.956821+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5438,"log_dups_size":0,"ondisk_log_size":5438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44512416,"num_objects":5434,"num_object_clones":0,"num_object_copies":21736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5436,"num_write_kb":43477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5314","reported_seq":5374,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.193358+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:00.193358+0000","last_peered":"2024-09-10T20:23:00.193358+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:00.193358+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43504800,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5297","reported_seq":5347,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.106724+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:00.106724+0000","last_peered":"2024-09-10T20:23:00.106724+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:00.106724+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43393024,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5297,"num_write_kb":42376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5331","reported_seq":5404,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.723710+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:22:59.723710+0000","last_peered":"2024-09-10T20:22:59.723710+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:22:59.723710+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43619488,"num_objects":5325,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":42612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3372}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5386","reported_seq":5458,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.871811+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:22:59.871811+0000","last_peered":"2024-09-10T20:22:59.871811+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:22:59.871811+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5386,"log_dups_size":0,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44078240,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5384,"num_write_kb":43060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5311","reported_seq":9273,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:00.114574+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:00.114574+0000","last_peered":"2024-09-10T20:23:00.114574+0000","last_clean":"2024-09-10T20:23:00.114574+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:00.114574+0000","last_undegraded":"2024-09-10T20:23:00.114574+0000","last_fullsized":"2024-09-10T20:23:00.114574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43416144,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1864,"num_write_kb":14912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5364","reported_seq":5449,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.495798+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:23:02.495798+0000","last_peered":"2024-09-10T20:23:02.495798+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:02.495798+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922592,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":42894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5302","reported_seq":5363,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.074605+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:00.074605+0000","last_peered":"2024-09-10T20:23:00.074605+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:00.074605+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5302,"log_dups_size":0,"ondisk_log_size":5302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43388656,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":42380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5309","reported_seq":5359,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.230358+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:00.230358+0000","last_peered":"2024-09-10T20:23:00.230358+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:00.230358+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43481680,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5308,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5238","reported_seq":5321,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.445471+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:02.445471+0000","last_peered":"2024-09-10T20:23:02.445471+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:02.445471+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42900048,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5314","reported_seq":5370,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:00.171621+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:00.171621+0000","last_peered":"2024-09-10T20:23:00.171621+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:00.171621+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43496608,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5525","reported_seq":5612,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.502635+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:02.502635+0000","last_peered":"2024-09-10T20:23:02.502635+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:02.502635+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45233312,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":44181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3560}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5310","reported_seq":5382,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:22:59.975783+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:22:59.975783+0000","last_peered":"2024-09-10T20:22:59.975783+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:22:59.975783+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5310,"log_dups_size":0,"ondisk_log_size":5310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43455648,"num_objects":5305,"num_object_clones":0,"num_object_copies":21220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5308,"num_write_kb":42452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":701013296,"num_objects":85577,"num_object_clones":0,"num_object_copies":342308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78608,"num_write_kb":628773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1820,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1299427328,"data_stored":1299427328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85636,"ondisk_log_size":85636,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602656,"kb_used_data":340496,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93769184,"statfs":{"total":96636764160,"available":96019644416,"internally_reserved":0,"allocated":348667904,"data_stored":348473531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":268431056},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627508,"kb_used_data":343652,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93744332,"statfs":{"total":96636764160,"available":95994195968,"internally_reserved":0,"allocated":351899648,"data_stored":351705275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473096,"kb_used_data":242872,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93898744,"statfs":{"total":96636764160,"available":96152313856,"internally_reserved":0,"allocated":248700928,"data_stored":248506555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602288,"kb_used_data":340192,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93769552,"statfs":{"total":96636764160,"available":96020021248,"internally_reserved":0,"allocated":348356608,"data_stored":348162235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,4,1,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":349270016,"data_stored":349270016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248397824,"data_stored":248397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352161792,"data_stored":352161792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":349597696,"data_stored":349597696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:05.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:05.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 204 20 86994 86974 3.33034 2.19531 0.0597534 0.0469015 2024-09-10T20:23:06.454 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:06.454 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:06.472 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-10T20:23:05.946379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709172528,"num_objects":86573,"num_object_clones":0,"num_object_copies":346292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79604,"num_write_kb":636741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86632,"ondisk_log_size":86632,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2363080,"kb_used_data":1280328,"kb_used_omap":14,"kb_used_meta":1082673,"kb_avail":375124280,"statfs":{"total":386547056640,"available":384127262720,"internally_reserved":0,"allocated":1311055872,"data_stored":1310278380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1108657718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,3,20,15,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004074"},"pg_stats":[{"pgid":"1.f","version":"36'5417","reported_seq":5465,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.200047+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:05.200047+0000","last_peered":"2024-09-10T20:23:05.200047+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:05.200047+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44376064,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":43336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5465","reported_seq":5585,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:03.473141+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:03.473141+0000","last_peered":"2024-09-10T20:23:03.473141+0000","last_clean":"2024-09-10T20:23:03.473141+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:03.473141+0000","last_undegraded":"2024-09-10T20:23:03.473141+0000","last_fullsized":"2024-09-10T20:23:03.473141+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44720128,"num_objects":5459,"num_object_clones":0,"num_object_copies":21836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1906,"num_write_kb":15248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5404","reported_seq":5488,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.544267+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:02.544267+0000","last_peered":"2024-09-10T20:23:02.544267+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:02.544267+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5404,"log_dups_size":0,"ondisk_log_size":5404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243536,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":43214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5525","reported_seq":9403,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.984285+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:23:04.984285+0000","last_peered":"2024-09-10T20:23:04.984285+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:23:04.984285+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45225120,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":44173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1905,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5389","reported_seq":5449,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.326900+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:05.326900+0000","last_peered":"2024-09-10T20:23:05.326900+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:05.326900+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44119200,"num_objects":5386,"num_object_clones":0,"num_object_copies":21544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5387,"num_write_kb":43093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5393","reported_seq":5443,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.270334+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:05.270334+0000","last_peered":"2024-09-10T20:23:05.270334+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:05.270334+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44179456,"num_objects":5393,"num_object_clones":0,"num_object_copies":21572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5411","reported_seq":5484,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.909821+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:04.909821+0000","last_peered":"2024-09-10T20:23:04.909821+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:04.909821+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44274848,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5409,"num_write_kb":43252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3452}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5474","reported_seq":5546,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.944977+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:04.944977+0000","last_peered":"2024-09-10T20:23:04.944977+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:04.944977+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799136,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5472,"num_write_kb":43764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5419","reported_seq":9381,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:05.258643+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:05.258643+0000","last_peered":"2024-09-10T20:23:05.258643+0000","last_clean":"2024-09-10T20:23:05.258643+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:05.258643+0000","last_undegraded":"2024-09-10T20:23:05.258643+0000","last_fullsized":"2024-09-10T20:23:05.258643+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44300880,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1972,"num_write_kb":15776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5364","reported_seq":5449,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.495798+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:23:02.495798+0000","last_peered":"2024-09-10T20:23:02.495798+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:02.495798+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922592,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":42894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5396","reported_seq":5459,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.147034+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:05.147034+0000","last_peered":"2024-09-10T20:23:05.147034+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:05.147034+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44158704,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5398","reported_seq":5450,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.938668+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:04.938668+0000","last_peered":"2024-09-10T20:23:04.938668+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:04.938668+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44210768,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5238","reported_seq":5321,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.445471+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:02.445471+0000","last_peered":"2024-09-10T20:23:02.445471+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:02.445471+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42900048,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5401","reported_seq":5457,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.195740+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:05.195740+0000","last_peered":"2024-09-10T20:23:05.195740+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:05.195740+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209312,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5525","reported_seq":5612,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.502635+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:02.502635+0000","last_peered":"2024-09-10T20:23:02.502635+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:02.502635+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45233312,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":44181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3560}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5413","reported_seq":5484,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.933941+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:04.933941+0000","last_peered":"2024-09-10T20:23:04.933941+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:04.933941+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5413,"log_dups_size":0,"ondisk_log_size":5413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44299424,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5411,"num_write_kb":43276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709172528,"num_objects":86573,"num_object_clones":0,"num_object_copies":346292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79604,"num_write_kb":636741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1311797248,"data_stored":1311797248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86632,"ondisk_log_size":86632,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632020,"kb_used_data":347652,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93739820,"statfs":{"total":96636764160,"available":95989575680,"internally_reserved":0,"allocated":355995648,"data_stored":355801275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":291172048},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627508,"kb_used_data":343652,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93744332,"statfs":{"total":96636764160,"available":95994195968,"internally_reserved":0,"allocated":351899648,"data_stored":351705275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473096,"kb_used_data":242872,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93898744,"statfs":{"total":96636764160,"available":96152313856,"internally_reserved":0,"allocated":248700928,"data_stored":248506555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630456,"kb_used_data":346152,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93741384,"statfs":{"total":96636764160,"available":95991177216,"internally_reserved":0,"allocated":354459648,"data_stored":354265275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355430400,"data_stored":355430400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248397824,"data_stored":248397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352161792,"data_stored":352161792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355807232,"data_stored":355807232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:06.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:06.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'5417", 'reported_seq': 5465, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:05.200047+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:05.200047+0000', 'last_peered': '2024-09-10T20:23:05.200047+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:05.200047+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5417, 'log_dups_size': 0, 'ondisk_log_size': 5417, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44376064, 'num_objects': 5417, 'num_object_clones': 0, 'num_object_copies': 21668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5417, 'num_write_kb': 43336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3451}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:06.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:06.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'5404", 'reported_seq': 5488, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:02.544267+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:02.544267+0000', 'last_peered': '2024-09-10T20:23:02.544267+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:02.544267+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5404, 'log_dups_size': 0, 'ondisk_log_size': 5404, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44243536, 'num_objects': 5401, 'num_object_clones': 0, 'num_object_copies': 21604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5403, 'num_write_kb': 43214, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3439}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:06.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'5525", 'reported_seq': 9403, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:04.984285+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:23:04.984285+0000', 'last_peered': '2024-09-10T20:23:04.984285+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:23:04.984285+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5525, 'log_dups_size': 0, 'ondisk_log_size': 5525, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45225120, 'num_objects': 5521, 'num_object_clones': 0, 'num_object_copies': 22084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 102, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5523, 'num_write_kb': 44173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1905, 'num_bytes_recovered': 15572992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'5389", 'reported_seq': 5449, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:05.326900+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:05.326900+0000', 'last_peered': '2024-09-10T20:23:05.326900+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:05.326900+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5389, 'log_dups_size': 0, 'ondisk_log_size': 5389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44119200, 'num_objects': 5386, 'num_object_clones': 0, 'num_object_copies': 21544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5386, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5387, 'num_write_kb': 43093, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3406}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'5393", 'reported_seq': 5443, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:05.270334+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:05.270334+0000', 'last_peered': '2024-09-10T20:23:05.270334+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:05.270334+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5393, 'log_dups_size': 0, 'ondisk_log_size': 5393, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44179456, 'num_objects': 5393, 'num_object_clones': 0, 'num_object_copies': 21572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5393, 'num_write_kb': 43144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3455}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'5411", 'reported_seq': 5484, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:04.909821+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:04.909821+0000', 'last_peered': '2024-09-10T20:23:04.909821+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:04.909821+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5411, 'log_dups_size': 0, 'ondisk_log_size': 5411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44274848, 'num_objects': 5405, 'num_object_clones': 0, 'num_object_copies': 21620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5409, 'num_write_kb': 43252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3452}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'5474", 'reported_seq': 5546, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:04.944977+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:04.944977+0000', 'last_peered': '2024-09-10T20:23:04.944977+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:04.944977+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5474, 'log_dups_size': 0, 'ondisk_log_size': 5474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44799136, 'num_objects': 5469, 'num_object_clones': 0, 'num_object_copies': 21876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5472, 'num_write_kb': 43764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'5364", 'reported_seq': 5449, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:02.495798+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:23:02.495798+0000', 'last_peered': '2024-09-10T20:23:02.495798+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:02.495798+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5364, 'log_dups_size': 0, 'ondisk_log_size': 5364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43922592, 'num_objects': 5362, 'num_object_clones': 0, 'num_object_copies': 21448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5362, 'num_write_kb': 42894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3459}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:06.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'5396", 'reported_seq': 5459, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:05.147034+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:05.147034+0000', 'last_peered': '2024-09-10T20:23:05.147034+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:05.147034+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5396, 'log_dups_size': 0, 'ondisk_log_size': 5396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44158704, 'num_objects': 5391, 'num_object_clones': 0, 'num_object_copies': 21564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5393, 'num_write_kb': 43132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3435}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'5398", 'reported_seq': 5450, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:04.938668+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:04.938668+0000', 'last_peered': '2024-09-10T20:23:04.938668+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:04.938668+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5398, 'log_dups_size': 0, 'ondisk_log_size': 5398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44210768, 'num_objects': 5397, 'num_object_clones': 0, 'num_object_copies': 21588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5397, 'num_write_kb': 43175, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'5238", 'reported_seq': 5321, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:02.445471+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:02.445471+0000', 'last_peered': '2024-09-10T20:23:02.445471+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:02.445471+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5238, 'log_dups_size': 0, 'ondisk_log_size': 5238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42900048, 'num_objects': 5237, 'num_object_clones': 0, 'num_object_copies': 20948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5237, 'num_write_kb': 41895, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'5401", 'reported_seq': 5457, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:05.195740+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:05.195740+0000', 'last_peered': '2024-09-10T20:23:05.195740+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:05.195740+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5401, 'log_dups_size': 0, 'ondisk_log_size': 5401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44209312, 'num_objects': 5397, 'num_object_clones': 0, 'num_object_copies': 21588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5399, 'num_write_kb': 43188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3422}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'5525", 'reported_seq': 5612, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:02.502635+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:02.502635+0000', 'last_peered': '2024-09-10T20:23:02.502635+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:02.502635+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5525, 'log_dups_size': 0, 'ondisk_log_size': 5525, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45233312, 'num_objects': 5522, 'num_object_clones': 0, 'num_object_copies': 22088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5523, 'num_write_kb': 44181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3560}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'5413", 'reported_seq': 5484, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:04.933941+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:04.933941+0000', 'last_peered': '2024-09-10T20:23:04.933941+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:04.933941+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5413, 'log_dups_size': 0, 'ondisk_log_size': 5413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44299424, 'num_objects': 5408, 'num_object_clones': 0, 'num_object_copies': 21632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5411, 'num_write_kb': 43276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:06.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:06.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 205 20 87279 87259 3.32495 2.22656 0.0403422 0.046978 2024-09-10T20:23:07.205 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:07.205 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25665/346292 objects degraded (7.411%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":275,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86573,"data_bytes":709172528,"bytes_used":2419793920,"bytes_avail":384127262720,"bytes_total":386547056640,"degraded_objects":25665,"degraded_total":346292,"degraded_ratio":0.07411375371074122,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":202683,"recovering_keys_per_sec":0,"num_objects_recovered":297,"num_bytes_recovered":2433024,"num_keys_recovered":0,"write_bytes_sec":2806172,"read_op_per_sec":0,"write_op_per_sec":342},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:07.229 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:07.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:07.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 206 20 87563 87543 3.31958 2.21875 0.0562014 0.0470579 2024-09-10T20:23:07.818 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:07.818 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:07.847 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-10T20:23:05.946379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709172528,"num_objects":86573,"num_object_clones":0,"num_object_copies":346292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79604,"num_write_kb":636741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86632,"ondisk_log_size":86632,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2363080,"kb_used_data":1280328,"kb_used_omap":14,"kb_used_meta":1082673,"kb_avail":375124280,"statfs":{"total":386547056640,"available":384127262720,"internally_reserved":0,"allocated":1311055872,"data_stored":1310278380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1108657718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,3,20,15,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004074"},"pg_stats":[{"pgid":"1.f","version":"36'5417","reported_seq":5465,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.200047+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:05.200047+0000","last_peered":"2024-09-10T20:23:05.200047+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:05.200047+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44376064,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":43336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5465","reported_seq":5585,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:03.473141+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:03.473141+0000","last_peered":"2024-09-10T20:23:03.473141+0000","last_clean":"2024-09-10T20:23:03.473141+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:03.473141+0000","last_undegraded":"2024-09-10T20:23:03.473141+0000","last_fullsized":"2024-09-10T20:23:03.473141+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44720128,"num_objects":5459,"num_object_clones":0,"num_object_copies":21836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1906,"num_write_kb":15248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5404","reported_seq":5488,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.544267+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:02.544267+0000","last_peered":"2024-09-10T20:23:02.544267+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:02.544267+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5404,"log_dups_size":0,"ondisk_log_size":5404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243536,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":43214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5525","reported_seq":9403,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.984285+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:23:04.984285+0000","last_peered":"2024-09-10T20:23:04.984285+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:23:04.984285+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45225120,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":44173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1905,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5389","reported_seq":5449,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.326900+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:05.326900+0000","last_peered":"2024-09-10T20:23:05.326900+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:05.326900+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44119200,"num_objects":5386,"num_object_clones":0,"num_object_copies":21544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5387,"num_write_kb":43093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5393","reported_seq":5443,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.270334+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:05.270334+0000","last_peered":"2024-09-10T20:23:05.270334+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:05.270334+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44179456,"num_objects":5393,"num_object_clones":0,"num_object_copies":21572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5411","reported_seq":5484,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.909821+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:04.909821+0000","last_peered":"2024-09-10T20:23:04.909821+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:04.909821+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44274848,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5409,"num_write_kb":43252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3452}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5474","reported_seq":5546,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.944977+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:04.944977+0000","last_peered":"2024-09-10T20:23:04.944977+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:04.944977+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799136,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5472,"num_write_kb":43764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5419","reported_seq":9381,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:05.258643+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:05.258643+0000","last_peered":"2024-09-10T20:23:05.258643+0000","last_clean":"2024-09-10T20:23:05.258643+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:05.258643+0000","last_undegraded":"2024-09-10T20:23:05.258643+0000","last_fullsized":"2024-09-10T20:23:05.258643+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44300880,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1972,"num_write_kb":15776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5364","reported_seq":5449,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.495798+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:23:02.495798+0000","last_peered":"2024-09-10T20:23:02.495798+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:02.495798+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922592,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":42894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5396","reported_seq":5459,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.147034+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:05.147034+0000","last_peered":"2024-09-10T20:23:05.147034+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:05.147034+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44158704,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5398","reported_seq":5450,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.938668+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:04.938668+0000","last_peered":"2024-09-10T20:23:04.938668+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:04.938668+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44210768,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5238","reported_seq":5321,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.445471+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:02.445471+0000","last_peered":"2024-09-10T20:23:02.445471+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:02.445471+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42900048,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5401","reported_seq":5457,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.195740+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:05.195740+0000","last_peered":"2024-09-10T20:23:05.195740+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:05.195740+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209312,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5525","reported_seq":5612,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:02.502635+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:02.502635+0000","last_peered":"2024-09-10T20:23:02.502635+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:02.502635+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45233312,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":44181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3560}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5413","reported_seq":5484,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.933941+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:04.933941+0000","last_peered":"2024-09-10T20:23:04.933941+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:04.933941+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5413,"log_dups_size":0,"ondisk_log_size":5413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44299424,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5411,"num_write_kb":43276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709172528,"num_objects":86573,"num_object_clones":0,"num_object_copies":346292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79604,"num_write_kb":636741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1311797248,"data_stored":1311797248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86632,"ondisk_log_size":86632,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632020,"kb_used_data":347652,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93739820,"statfs":{"total":96636764160,"available":95989575680,"internally_reserved":0,"allocated":355995648,"data_stored":355801275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":291172048},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627508,"kb_used_data":343652,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93744332,"statfs":{"total":96636764160,"available":95994195968,"internally_reserved":0,"allocated":351899648,"data_stored":351705275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473096,"kb_used_data":242872,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93898744,"statfs":{"total":96636764160,"available":96152313856,"internally_reserved":0,"allocated":248700928,"data_stored":248506555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630456,"kb_used_data":346152,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93741384,"statfs":{"total":96636764160,"available":95991177216,"internally_reserved":0,"allocated":354459648,"data_stored":354265275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355430400,"data_stored":355430400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248397824,"data_stored":248397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352161792,"data_stored":352161792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355807232,"data_stored":355807232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:08.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 207 20 87882 87862 3.31558 2.49219 0.0434873 0.0471155 2024-09-10T20:23:09.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 208 20 88211 88191 3.312 2.57031 0.0404384 0.0471658 2024-09-10T20:23:10.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 209 20 88532 88512 3.30815 2.50781 0.0862395 0.0472195 2024-09-10T20:23:10.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:11.426 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:11.426 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:11.448 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-10T20:23:09.947966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":713153840,"num_objects":87059,"num_object_clones":0,"num_object_copies":348236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80090,"num_write_kb":640629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87118,"ondisk_log_size":87118,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2383764,"kb_used_data":1292692,"kb_used_omap":14,"kb_used_meta":1090993,"kb_avail":375103596,"statfs":{"total":386547056640,"available":384106082304,"internally_reserved":0,"allocated":1323716608,"data_stored":1322939116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1117177398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,8,1,21,20,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25763840,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3145,"num_write_kb":25160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":1499136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004318"},"pg_stats":[{"pgid":"1.f","version":"36'5417","reported_seq":5465,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.200047+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:05.200047+0000","last_peered":"2024-09-10T20:23:05.200047+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:05.200047+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44376064,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":43336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5571","reported_seq":5693,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:08.573781+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:08.573781+0000","last_peered":"2024-09-10T20:23:08.573781+0000","last_clean":"2024-09-10T20:23:08.573781+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:08.573781+0000","last_undegraded":"2024-09-10T20:23:08.573781+0000","last_fullsized":"2024-09-10T20:23:08.573781+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45588480,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":16096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5485","reported_seq":5569,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.551217+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:07.551217+0000","last_peered":"2024-09-10T20:23:07.551217+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:07.551217+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":43862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3520}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5525","reported_seq":9403,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.984285+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:23:04.984285+0000","last_peered":"2024-09-10T20:23:04.984285+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:23:04.984285+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45225120,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":44173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1905,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5389","reported_seq":5449,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.326900+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:05.326900+0000","last_peered":"2024-09-10T20:23:05.326900+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:05.326900+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44119200,"num_objects":5386,"num_object_clones":0,"num_object_copies":21544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5387,"num_write_kb":43093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5393","reported_seq":5443,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.270334+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:05.270334+0000","last_peered":"2024-09-10T20:23:05.270334+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:05.270334+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44179456,"num_objects":5393,"num_object_clones":0,"num_object_copies":21572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5411","reported_seq":5484,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.909821+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:04.909821+0000","last_peered":"2024-09-10T20:23:04.909821+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:04.909821+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44274848,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5409,"num_write_kb":43252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3452}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5474","reported_seq":5546,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.944977+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:04.944977+0000","last_peered":"2024-09-10T20:23:04.944977+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:04.944977+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799136,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5472,"num_write_kb":43764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5419","reported_seq":9381,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:05.258643+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:05.258643+0000","last_peered":"2024-09-10T20:23:05.258643+0000","last_clean":"2024-09-10T20:23:05.258643+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:05.258643+0000","last_undegraded":"2024-09-10T20:23:05.258643+0000","last_fullsized":"2024-09-10T20:23:05.258643+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44300880,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1972,"num_write_kb":15776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5461","reported_seq":5547,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.605868+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:23:07.605868+0000","last_peered":"2024-09-10T20:23:07.605868+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:07.605868+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44717216,"num_objects":5459,"num_object_clones":0,"num_object_copies":21836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5459,"num_write_kb":43670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5396","reported_seq":5459,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.147034+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:05.147034+0000","last_peered":"2024-09-10T20:23:05.147034+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:05.147034+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44158704,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5398","reported_seq":5450,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.938668+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:04.938668+0000","last_peered":"2024-09-10T20:23:04.938668+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:04.938668+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44210768,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5338","reported_seq":5421,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.482773+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:07.482773+0000","last_peered":"2024-09-10T20:23:07.482773+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:07.482773+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43719248,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5337,"num_write_kb":42695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5401","reported_seq":5457,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:05.195740+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:05.195740+0000","last_peered":"2024-09-10T20:23:05.195740+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:05.195740+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209312,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5627","reported_seq":5714,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.612293+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:07.612293+0000","last_peered":"2024-09-10T20:23:07.612293+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:07.612293+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46068896,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":44997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5413","reported_seq":5484,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:04.933941+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:04.933941+0000","last_peered":"2024-09-10T20:23:04.933941+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:04.933941+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5413,"log_dups_size":0,"ondisk_log_size":5413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44299424,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5411,"num_write_kb":43276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":713153840,"num_objects":87059,"num_object_clones":0,"num_object_copies":348236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80090,"num_write_kb":640629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":15572992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1324675072,"data_stored":1324675072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87118,"ondisk_log_size":87118,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632020,"kb_used_data":347652,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93739820,"statfs":{"total":96636764160,"available":95989575680,"internally_reserved":0,"allocated":355995648,"data_stored":355801275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":291172048},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637688,"kb_used_data":349672,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93734152,"statfs":{"total":96636764160,"available":95983771648,"internally_reserved":0,"allocated":358064128,"data_stored":357869755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483600,"kb_used_data":249216,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93888240,"statfs":{"total":96636764160,"available":96141557760,"internally_reserved":0,"allocated":255197184,"data_stored":255002811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,0,4,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630456,"kb_used_data":346152,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93741384,"statfs":{"total":96636764160,"available":95991177216,"internally_reserved":0,"allocated":354459648,"data_stored":354265275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355430400,"data_stored":355430400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254984192,"data_stored":254984192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":358453248,"data_stored":358453248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355807232,"data_stored":355807232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:11.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:11.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 210 20 88811 88791 3.30278 2.17969 0.099934 0.0472966 2024-09-10T20:23:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:12.074 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:12.100 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-10T20:23:11.948749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":721886512,"num_objects":88125,"num_object_clones":0,"num_object_copies":352500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81156,"num_write_kb":649157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":16261120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88184,"ondisk_log_size":88184,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2440128,"kb_used_data":1304640,"kb_used_omap":14,"kb_used_meta":1135409,"kb_avail":375047232,"statfs":{"total":386547056640,"available":384048365568,"internally_reserved":0,"allocated":1335951360,"data_stored":1335173868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1162659382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,9,7,10,25,25,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33816576,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":33024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":2187264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004279"},"pg_stats":[{"pgid":"1.f","version":"36'5529","reported_seq":5579,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.278748+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:10.278748+0000","last_peered":"2024-09-10T20:23:10.278748+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:10.278748+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5529,"log_dups_size":0,"ondisk_log_size":5529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45293568,"num_objects":5529,"num_object_clones":0,"num_object_copies":22116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5571","reported_seq":5693,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:08.573781+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:08.573781+0000","last_peered":"2024-09-10T20:23:08.573781+0000","last_clean":"2024-09-10T20:23:08.573781+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:08.573781+0000","last_undegraded":"2024-09-10T20:23:08.573781+0000","last_fullsized":"2024-09-10T20:23:08.573781+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45588480,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":16096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'5485","reported_seq":5569,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.551217+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:07.551217+0000","last_peered":"2024-09-10T20:23:07.551217+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:07.551217+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":43862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3520}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5627","reported_seq":9672,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.046220+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:23:10.046220+0000","last_peered":"2024-09-10T20:23:10.046220+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:23:10.046220+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":44989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1989,"num_bytes_recovered":16261120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5483","reported_seq":5543,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.173825+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:10.173825+0000","last_peered":"2024-09-10T20:23:10.173825+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:10.173825+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44889248,"num_objects":5480,"num_object_clones":0,"num_object_copies":21920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3500}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5474","reported_seq":5524,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.356009+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:10.356009+0000","last_peered":"2024-09-10T20:23:10.356009+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:10.356009+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44843008,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5517","reported_seq":5588,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:09.852848+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:09.852848+0000","last_peered":"2024-09-10T20:23:09.852848+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:09.852848+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5517,"log_dups_size":0,"ondisk_log_size":5517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45143200,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":44100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3558}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5575","reported_seq":5647,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:09.826056+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:09.826056+0000","last_peered":"2024-09-10T20:23:09.826056+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:09.826056+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45626528,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5506","reported_seq":9467,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:10.292112+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:10.292112+0000","last_peered":"2024-09-10T20:23:10.292112+0000","last_clean":"2024-09-10T20:23:10.292112+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:10.292112+0000","last_undegraded":"2024-09-10T20:23:10.292112+0000","last_fullsized":"2024-09-10T20:23:10.292112+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45013584,"num_objects":5495,"num_object_clones":0,"num_object_copies":21980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'5461","reported_seq":5547,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.605868+0000","last_change":"2024-09-10T20:20:16.103340+0000","last_active":"2024-09-10T20:23:07.605868+0000","last_peered":"2024-09-10T20:23:07.605868+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:07.605868+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44717216,"num_objects":5459,"num_object_clones":0,"num_object_copies":21836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5459,"num_write_kb":43670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5480","reported_seq":5543,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.343010+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:10.343010+0000","last_peered":"2024-09-10T20:23:10.343010+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:10.343010+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44846832,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5477,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3519}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5498","reported_seq":5550,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.295748+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:10.295748+0000","last_peered":"2024-09-10T20:23:10.295748+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:10.295748+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45029968,"num_objects":5497,"num_object_clones":0,"num_object_copies":21988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5497,"num_write_kb":43975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3519}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'5338","reported_seq":5421,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.482773+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:07.482773+0000","last_peered":"2024-09-10T20:23:07.482773+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:07.482773+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43719248,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5337,"num_write_kb":42695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5501","reported_seq":5560,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.169435+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:10.169435+0000","last_peered":"2024-09-10T20:23:10.169435+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:10.169435+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5501,"log_dups_size":0,"ondisk_log_size":5501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45028512,"num_objects":5497,"num_object_clones":0,"num_object_copies":21988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5499,"num_write_kb":43988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'5627","reported_seq":5714,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:07.612293+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:07.612293+0000","last_peered":"2024-09-10T20:23:07.612293+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:07.612293+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46068896,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":44997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5512","reported_seq":5584,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:09.950328+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:09.950328+0000","last_peered":"2024-09-10T20:23:09.950328+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:09.950328+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110432,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":44068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":721886512,"num_objects":88125,"num_object_clones":0,"num_object_copies":352500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81156,"num_write_kb":649157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":16261120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337131008,"data_stored":1337131008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88184,"ondisk_log_size":88184,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678232,"kb_used_data":353608,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93693608,"statfs":{"total":96636764160,"available":95942254592,"internally_reserved":0,"allocated":362094592,"data_stored":361900219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,0,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637688,"kb_used_data":349672,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93734152,"statfs":{"total":96636764160,"available":95983771648,"internally_reserved":0,"allocated":358064128,"data_stored":357869755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483600,"kb_used_data":249216,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93888240,"statfs":{"total":96636764160,"available":96141557760,"internally_reserved":0,"allocated":255197184,"data_stored":255002811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,0,4,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640608,"kb_used_data":352144,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93731232,"statfs":{"total":96636764160,"available":95980781568,"internally_reserved":0,"allocated":360595456,"data_stored":360401083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,5,5,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361619456,"data_stored":361619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254984192,"data_stored":254984192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":358453248,"data_stored":358453248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362074112,"data_stored":362074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'5529", 'reported_seq': 5579, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:10.278748+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:10.278748+0000', 'last_peered': '2024-09-10T20:23:10.278748+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:10.278748+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5529, 'log_dups_size': 0, 'ondisk_log_size': 5529, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45293568, 'num_objects': 5529, 'num_object_clones': 0, 'num_object_copies': 22116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5529, 'num_write_kb': 44232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3563}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'5485", 'reported_seq': 5569, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:07.551217+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:07.551217+0000', 'last_peered': '2024-09-10T20:23:07.551217+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:07.551217+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5485, 'log_dups_size': 0, 'ondisk_log_size': 5485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44907088, 'num_objects': 5482, 'num_object_clones': 0, 'num_object_copies': 21928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5484, 'num_write_kb': 43862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3520}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'5627", 'reported_seq': 9672, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:10.046220+0000', 'last_change': '2024-09-10T20:21:29.738267+0000', 'last_active': '2024-09-10T20:23:10.046220+0000', 'last_peered': '2024-09-10T20:23:10.046220+0000', 'last_clean': '2024-09-10T20:19:05.822050+0000', 'last_became_active': '2024-09-10T20:20:16.137758+0000', 'last_became_peered': '2024-09-10T20:20:16.137758+0000', 'last_unstale': '2024-09-10T20:23:10.046220+0000', 'last_undegraded': '2024-09-10T20:20:15.482243+0000', 'last_fullsized': '2024-09-10T20:20:15.455278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5627, 'log_dups_size': 0, 'ondisk_log_size': 5627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46060704, 'num_objects': 5623, 'num_object_clones': 0, 'num_object_copies': 22492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5625, 'num_write_kb': 44989, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1989, 'num_bytes_recovered': 16261120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5605}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:12.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'5483", 'reported_seq': 5543, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:10.173825+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:10.173825+0000', 'last_peered': '2024-09-10T20:23:10.173825+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:10.173825+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5483, 'log_dups_size': 0, 'ondisk_log_size': 5483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44889248, 'num_objects': 5480, 'num_object_clones': 0, 'num_object_copies': 21920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5481, 'num_write_kb': 43845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3500}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'5474", 'reported_seq': 5524, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:10.356009+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:10.356009+0000', 'last_peered': '2024-09-10T20:23:10.356009+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:10.356009+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5474, 'log_dups_size': 0, 'ondisk_log_size': 5474, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44843008, 'num_objects': 5474, 'num_object_clones': 0, 'num_object_copies': 21896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5474, 'num_write_kb': 43792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3536}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "36'5517", 'reported_seq': 5588, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:09.852848+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:09.852848+0000', 'last_peered': '2024-09-10T20:23:09.852848+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:09.852848+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5517, 'log_dups_size': 0, 'ondisk_log_size': 5517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45143200, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 22044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5515, 'num_write_kb': 44100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3558}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'5575", 'reported_seq': 5647, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:09.826056+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:09.826056+0000', 'last_peered': '2024-09-10T20:23:09.826056+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:09.826056+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5575, 'log_dups_size': 0, 'ondisk_log_size': 5575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45626528, 'num_objects': 5570, 'num_object_clones': 0, 'num_object_copies': 22280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5573, 'num_write_kb': 44572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'5461", 'reported_seq': 5547, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:07.605868+0000', 'last_change': '2024-09-10T20:20:16.103340+0000', 'last_active': '2024-09-10T20:23:07.605868+0000', 'last_peered': '2024-09-10T20:23:07.605868+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:07.605868+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5461, 'log_dups_size': 0, 'ondisk_log_size': 5461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44717216, 'num_objects': 5459, 'num_object_clones': 0, 'num_object_copies': 21836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5459, 'num_write_kb': 43670, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3556}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'5480", 'reported_seq': 5543, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:10.343010+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:10.343010+0000', 'last_peered': '2024-09-10T20:23:10.343010+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:10.343010+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5480, 'log_dups_size': 0, 'ondisk_log_size': 5480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44846832, 'num_objects': 5475, 'num_object_clones': 0, 'num_object_copies': 21900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5477, 'num_write_kb': 43804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3519}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'5498", 'reported_seq': 5550, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:10.295748+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:10.295748+0000', 'last_peered': '2024-09-10T20:23:10.295748+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:10.295748+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5498, 'log_dups_size': 0, 'ondisk_log_size': 5498, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45029968, 'num_objects': 5497, 'num_object_clones': 0, 'num_object_copies': 21988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5497, 'num_write_kb': 43975, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'5338", 'reported_seq': 5421, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:07.482773+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:07.482773+0000', 'last_peered': '2024-09-10T20:23:07.482773+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:07.482773+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5338, 'log_dups_size': 0, 'ondisk_log_size': 5338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43719248, 'num_objects': 5337, 'num_object_clones': 0, 'num_object_copies': 21348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5337, 'num_write_kb': 42695, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'5501", 'reported_seq': 5560, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:10.169435+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:10.169435+0000', 'last_peered': '2024-09-10T20:23:10.169435+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:10.169435+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5501, 'log_dups_size': 0, 'ondisk_log_size': 5501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45028512, 'num_objects': 5497, 'num_object_clones': 0, 'num_object_copies': 21988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5499, 'num_write_kb': 43988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "36'5627", 'reported_seq': 5714, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:07.612293+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:07.612293+0000', 'last_peered': '2024-09-10T20:23:07.612293+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:07.612293+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5627, 'log_dups_size': 0, 'ondisk_log_size': 5627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46068896, 'num_objects': 5624, 'num_object_clones': 0, 'num_object_copies': 22496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5625, 'num_write_kb': 44997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:12.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'5512", 'reported_seq': 5584, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:09.950328+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:09.950328+0000', 'last_peered': '2024-09-10T20:23:09.950328+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:09.950328+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5512, 'log_dups_size': 0, 'ondisk_log_size': 5512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45110432, 'num_objects': 5507, 'num_object_clones': 0, 'num_object_copies': 22028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5510, 'num_write_kb': 44068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:12.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:12.798 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 211 20 89047 89027 3.29586 1.84375 0.0285651 0.0473611 2024-09-10T20:23:12.818 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:12.819 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25581/352500 objects degraded (7.257%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":281,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88125,"data_bytes":721886512,"bytes_used":2498691072,"bytes_avail":384048365568,"bytes_total":386547056640,"degraded_objects":25581,"degraded_total":352500,"degraded_ratio":0.072570212765957445,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":182207,"recovering_keys_per_sec":0,"num_objects_recovered":267,"num_bytes_recovered":2187264,"num_keys_recovered":0,"write_bytes_sec":2817043,"read_op_per_sec":0,"write_op_per_sec":343},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:12.847 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:12.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:13.434 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:13.461 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-10T20:23:13.110422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":724999472,"num_objects":88505,"num_object_clones":0,"num_object_copies":354020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81536,"num_write_kb":652197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88564,"ondisk_log_size":88564,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2504684,"kb_used_data":1310888,"kb_used_omap":14,"kb_used_meta":1193713,"kb_avail":374982676,"statfs":{"total":386547056640,"available":383982260224,"internally_reserved":0,"allocated":1342349312,"data_stored":1341567177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1222362678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,10,8,11,21,22,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23986176,"num_objects":2928,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":23424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.164713"},"pg_stats":[{"pgid":"1.f","version":"36'5529","reported_seq":5579,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.278748+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:10.278748+0000","last_peered":"2024-09-10T20:23:10.278748+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:10.278748+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5529,"log_dups_size":0,"ondisk_log_size":5529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45293568,"num_objects":5529,"num_object_clones":0,"num_object_copies":22116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'5571","reported_seq":5693,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:08.573781+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:08.573781+0000","last_peered":"2024-09-10T20:23:08.573781+0000","last_clean":"2024-09-10T20:23:08.573781+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:08.573781+0000","last_undegraded":"2024-09-10T20:23:08.573781+0000","last_fullsized":"2024-09-10T20:23:08.573781+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45588480,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":16096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'5579","reported_seq":5665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.517152+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:12.517152+0000","last_peered":"2024-09-10T20:23:12.517152+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:12.517152+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45677136,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3614}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'5627","reported_seq":9672,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.046220+0000","last_change":"2024-09-10T20:21:29.738267+0000","last_active":"2024-09-10T20:23:10.046220+0000","last_peered":"2024-09-10T20:23:10.046220+0000","last_clean":"2024-09-10T20:19:05.822050+0000","last_became_active":"2024-09-10T20:20:16.137758+0000","last_became_peered":"2024-09-10T20:20:16.137758+0000","last_unstale":"2024-09-10T20:23:10.046220+0000","last_undegraded":"2024-09-10T20:20:15.482243+0000","last_fullsized":"2024-09-10T20:20:15.455278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":44989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1989,"num_bytes_recovered":16261120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'5483","reported_seq":5543,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.173825+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:10.173825+0000","last_peered":"2024-09-10T20:23:10.173825+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:10.173825+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44889248,"num_objects":5480,"num_object_clones":0,"num_object_copies":21920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3500}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'5474","reported_seq":5524,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.356009+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:10.356009+0000","last_peered":"2024-09-10T20:23:10.356009+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:10.356009+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44843008,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'5517","reported_seq":5588,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:09.852848+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:09.852848+0000","last_peered":"2024-09-10T20:23:09.852848+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:09.852848+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5517,"log_dups_size":0,"ondisk_log_size":5517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45143200,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":44100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3558}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'5575","reported_seq":5647,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:09.826056+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:09.826056+0000","last_peered":"2024-09-10T20:23:09.826056+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:09.826056+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45626528,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'5506","reported_seq":9467,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-10T20:23:10.292112+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:10.292112+0000","last_peered":"2024-09-10T20:23:10.292112+0000","last_clean":"2024-09-10T20:23:10.292112+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:10.292112+0000","last_undegraded":"2024-09-10T20:23:10.292112+0000","last_fullsized":"2024-09-10T20:23:10.292112+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45013584,"num_objects":5495,"num_object_clones":0,"num_object_copies":21980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'5553","reported_seq":5688,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.609386+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:12.609386+0000","last_peered":"2024-09-10T20:23:12.609386+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:12.609386+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45470880,"num_objects":5551,"num_object_clones":0,"num_object_copies":22204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":44406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":188416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3671}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'5480","reported_seq":5543,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.343010+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:10.343010+0000","last_peered":"2024-09-10T20:23:10.343010+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:10.343010+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44846832,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5477,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3519}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'5498","reported_seq":5550,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.295748+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:10.295748+0000","last_peered":"2024-09-10T20:23:10.295748+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:10.295748+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45029968,"num_objects":5497,"num_object_clones":0,"num_object_copies":21988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5497,"num_write_kb":43975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3519}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'5441","reported_seq":5526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.610522+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:12.610522+0000","last_peered":"2024-09-10T20:23:12.610522+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:12.610522+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44563024,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5440,"num_write_kb":43519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'5501","reported_seq":5560,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:10.169435+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:10.169435+0000","last_peered":"2024-09-10T20:23:10.169435+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:10.169435+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5501,"log_dups_size":0,"ondisk_log_size":5501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45028512,"num_objects":5497,"num_object_clones":0,"num_object_copies":21988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5499,"num_write_kb":43988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'5718","reported_seq":5807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.479108+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:12.479108+0000","last_peered":"2024-09-10T20:23:12.479108+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:12.479108+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46814368,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"36'5512","reported_seq":5584,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:09.950328+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:09.950328+0000","last_peered":"2024-09-10T20:23:09.950328+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:09.950328+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110432,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":44068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":724999472,"num_objects":88505,"num_object_clones":0,"num_object_copies":354020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81536,"num_write_kb":652197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1343238144,"data_stored":1343238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88564,"ondisk_log_size":88564,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678232,"kb_used_data":353608,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93693608,"statfs":{"total":96636764160,"available":95942254592,"internally_reserved":0,"allocated":362094592,"data_stored":361900219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,0,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702244,"kb_used_data":355920,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93669596,"statfs":{"total":96636764160,"available":95917666304,"internally_reserved":0,"allocated":364462080,"data_stored":364263064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483600,"kb_used_data":249216,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93888240,"statfs":{"total":96636764160,"available":96141557760,"internally_reserved":0,"allocated":255197184,"data_stored":255002811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,0,4,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640608,"kb_used_data":352144,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93731232,"statfs":{"total":96636764160,"available":95980781568,"internally_reserved":0,"allocated":360595456,"data_stored":360401083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,5,5,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361619456,"data_stored":361619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254984192,"data_stored":254984192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364560384,"data_stored":364560384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362074112,"data_stored":362074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:13.799 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 212 20 89300 89280 3.28964 1.97656 0.0532826 0.0474833 2024-09-10T20:23:14.799 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 213 20 89577 89557 3.28435 2.16406 0.0443375 0.0475581 2024-09-10T20:23:15.799 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 214 20 89884 89864 3.28021 2.39844 0.025084 0.0476209 2024-09-10T20:23:16.462 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:16.799 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 215 20 90170 90150 3.27534 2.23438 0.0378883 0.0476927 2024-09-10T20:23:17.137 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:17.138 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:17.173 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-10T20:23:15.950039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733183280,"num_objects":89504,"num_object_clones":0,"num_object_copies":358016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82535,"num_write_kb":660189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16596992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89563,"ondisk_log_size":89563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2606600,"kb_used_data":1328120,"kb_used_omap":14,"kb_used_meta":1278385,"kb_avail":374880760,"statfs":{"total":386547056640,"available":383877898240,"internally_reserved":0,"allocated":1359994880,"data_stored":1359184824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1309066806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,7,10,10,16,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20897792,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2551,"num_write_kb":20408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002926"},"pg_stats":[{"pgid":"1.f","version":"38'5624","reported_seq":5678,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.395734+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:15.395734+0000","last_peered":"2024-09-10T20:23:15.395734+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:15.395734+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":44992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5665","reported_seq":5791,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:13.595664+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:13.595664+0000","last_peered":"2024-09-10T20:23:13.595664+0000","last_clean":"2024-09-10T20:23:13.595664+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:13.595664+0000","last_undegraded":"2024-09-10T20:23:13.595664+0000","last_fullsized":"2024-09-10T20:23:13.595664+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46358528,"num_objects":5659,"num_object_clones":0,"num_object_copies":22636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'5579","reported_seq":5665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.517152+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:12.517152+0000","last_peered":"2024-09-10T20:23:12.517152+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:12.517152+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45677136,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3614}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5692","reported_seq":9795,"reported_epoch":38,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:23:13.631409+0000","last_change":"2024-09-10T20:23:13.591911+0000","last_active":"2024-09-10T20:23:13.631409+0000","last_peered":"2024-09-10T20:23:13.631409+0000","last_clean":"2024-09-10T20:23:13.631409+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:13.631409+0000","last_undegraded":"2024-09-10T20:23:13.631409+0000","last_fullsized":"2024-09-10T20:23:13.631409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":99,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46593184,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5690,"num_write_kb":45509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2007,"num_bytes_recovered":16408576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5568","reported_seq":5632,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.343879+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:15.343879+0000","last_peered":"2024-09-10T20:23:15.343879+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:15.343879+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45585568,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5566,"num_write_kb":44525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5558","reported_seq":5612,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.157786+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:15.157786+0000","last_peered":"2024-09-10T20:23:15.157786+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:15.157786+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45531136,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":44464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5592","reported_seq":5669,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:14.766429+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:14.766429+0000","last_peered":"2024-09-10T20:23:14.766429+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:14.766429+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45757600,"num_objects":5586,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":44700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3633}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5655","reported_seq":5731,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.013526+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:15.013526+0000","last_peered":"2024-09-10T20:23:15.013526+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:15.013526+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46281888,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":45212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3607}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5592","reported_seq":9558,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:15.313824+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:15.313824+0000","last_peered":"2024-09-10T20:23:15.313824+0000","last_clean":"2024-09-10T20:23:15.313824+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:15.313824+0000","last_undegraded":"2024-09-10T20:23:15.313824+0000","last_fullsized":"2024-09-10T20:23:15.313824+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45718096,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":17160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'5553","reported_seq":5688,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.609386+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:12.609386+0000","last_peered":"2024-09-10T20:23:12.609386+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:12.609386+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45470880,"num_objects":5551,"num_object_clones":0,"num_object_copies":22204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":44406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":188416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3671}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5563","reported_seq":5630,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.378481+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:15.378481+0000","last_peered":"2024-09-10T20:23:15.378481+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:15.378481+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45526768,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5587","reported_seq":5642,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.375263+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:15.375263+0000","last_peered":"2024-09-10T20:23:15.375263+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:15.375263+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45759056,"num_objects":5586,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":44687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'5441","reported_seq":5526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.610522+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:12.610522+0000","last_peered":"2024-09-10T20:23:12.610522+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:12.610522+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44563024,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5440,"num_write_kb":43519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5583","reported_seq":5646,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.160043+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:15.160043+0000","last_peered":"2024-09-10T20:23:15.160043+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:15.160043+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700256,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3604}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'5718","reported_seq":5807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.479108+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:12.479108+0000","last_peered":"2024-09-10T20:23:12.479108+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:12.479108+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46814368,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5593","reported_seq":5669,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:14.980966+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:14.980966+0000","last_peered":"2024-09-10T20:23:14.980966+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:14.980966+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5593,"log_dups_size":0,"ondisk_log_size":5593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45773984,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5591,"num_write_kb":44716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733183280,"num_objects":89504,"num_object_clones":0,"num_object_copies":358016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82535,"num_write_kb":660189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16596992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1360748544,"data_stored":1360748544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89563,"ondisk_log_size":89563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705896,"kb_used_data":359060,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93665944,"statfs":{"total":96636764160,"available":95913926656,"internally_reserved":0,"allocated":367677440,"data_stored":367473760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,5,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702244,"kb_used_data":355920,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93669596,"statfs":{"total":96636764160,"available":95917666304,"internally_reserved":0,"allocated":364462080,"data_stored":364263064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493888,"kb_used_data":255336,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93877952,"statfs":{"total":96636764160,"available":96131022848,"internally_reserved":0,"allocated":261464064,"data_stored":261260384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704572,"kb_used_data":357804,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93667268,"statfs":{"total":96636764160,"available":95915282432,"internally_reserved":0,"allocated":366391296,"data_stored":366187616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367243264,"data_stored":367243264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261230592,"data_stored":261230592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364560384,"data_stored":364560384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367714304,"data_stored":367714304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:17.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:17.710 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:17.737 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-10T20:23:15.950039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733183280,"num_objects":89504,"num_object_clones":0,"num_object_copies":358016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82535,"num_write_kb":660189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16596992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89563,"ondisk_log_size":89563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2606600,"kb_used_data":1328120,"kb_used_omap":14,"kb_used_meta":1278385,"kb_avail":374880760,"statfs":{"total":386547056640,"available":383877898240,"internally_reserved":0,"allocated":1359994880,"data_stored":1359184824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1309066806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,7,10,10,16,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20897792,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2551,"num_write_kb":20408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002926"},"pg_stats":[{"pgid":"1.f","version":"38'5624","reported_seq":5678,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.395734+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:15.395734+0000","last_peered":"2024-09-10T20:23:15.395734+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:15.395734+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":44992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5665","reported_seq":5791,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:13.595664+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:13.595664+0000","last_peered":"2024-09-10T20:23:13.595664+0000","last_clean":"2024-09-10T20:23:13.595664+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:13.595664+0000","last_undegraded":"2024-09-10T20:23:13.595664+0000","last_fullsized":"2024-09-10T20:23:13.595664+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46358528,"num_objects":5659,"num_object_clones":0,"num_object_copies":22636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'5579","reported_seq":5665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.517152+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:12.517152+0000","last_peered":"2024-09-10T20:23:12.517152+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:12.517152+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45677136,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3614}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5692","reported_seq":9795,"reported_epoch":38,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:23:13.631409+0000","last_change":"2024-09-10T20:23:13.591911+0000","last_active":"2024-09-10T20:23:13.631409+0000","last_peered":"2024-09-10T20:23:13.631409+0000","last_clean":"2024-09-10T20:23:13.631409+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:13.631409+0000","last_undegraded":"2024-09-10T20:23:13.631409+0000","last_fullsized":"2024-09-10T20:23:13.631409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":99,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46593184,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5690,"num_write_kb":45509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2007,"num_bytes_recovered":16408576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5568","reported_seq":5632,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.343879+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:15.343879+0000","last_peered":"2024-09-10T20:23:15.343879+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:15.343879+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45585568,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5566,"num_write_kb":44525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5558","reported_seq":5612,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.157786+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:15.157786+0000","last_peered":"2024-09-10T20:23:15.157786+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:15.157786+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45531136,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":44464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5592","reported_seq":5669,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:14.766429+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:14.766429+0000","last_peered":"2024-09-10T20:23:14.766429+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:14.766429+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45757600,"num_objects":5586,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":44700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3633}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5655","reported_seq":5731,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.013526+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:15.013526+0000","last_peered":"2024-09-10T20:23:15.013526+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:15.013526+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46281888,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":45212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3607}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5592","reported_seq":9558,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:15.313824+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:15.313824+0000","last_peered":"2024-09-10T20:23:15.313824+0000","last_clean":"2024-09-10T20:23:15.313824+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:15.313824+0000","last_undegraded":"2024-09-10T20:23:15.313824+0000","last_fullsized":"2024-09-10T20:23:15.313824+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45718096,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":17160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'5553","reported_seq":5688,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.609386+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:12.609386+0000","last_peered":"2024-09-10T20:23:12.609386+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:12.609386+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45470880,"num_objects":5551,"num_object_clones":0,"num_object_copies":22204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":44406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":188416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3671}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5563","reported_seq":5630,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.378481+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:15.378481+0000","last_peered":"2024-09-10T20:23:15.378481+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:15.378481+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45526768,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5587","reported_seq":5642,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.375263+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:15.375263+0000","last_peered":"2024-09-10T20:23:15.375263+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:15.375263+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45759056,"num_objects":5586,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":44687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'5441","reported_seq":5526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.610522+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:12.610522+0000","last_peered":"2024-09-10T20:23:12.610522+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:12.610522+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44563024,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5440,"num_write_kb":43519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5583","reported_seq":5646,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.160043+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:15.160043+0000","last_peered":"2024-09-10T20:23:15.160043+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:15.160043+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700256,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3604}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'5718","reported_seq":5807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:12.479108+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:12.479108+0000","last_peered":"2024-09-10T20:23:12.479108+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:12.479108+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46814368,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5593","reported_seq":5669,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:14.980966+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:14.980966+0000","last_peered":"2024-09-10T20:23:14.980966+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:14.980966+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5593,"log_dups_size":0,"ondisk_log_size":5593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45773984,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5591,"num_write_kb":44716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733183280,"num_objects":89504,"num_object_clones":0,"num_object_copies":358016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82535,"num_write_kb":660189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16596992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1360748544,"data_stored":1360748544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89563,"ondisk_log_size":89563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705896,"kb_used_data":359060,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93665944,"statfs":{"total":96636764160,"available":95913926656,"internally_reserved":0,"allocated":367677440,"data_stored":367473760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,5,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702244,"kb_used_data":355920,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93669596,"statfs":{"total":96636764160,"available":95917666304,"internally_reserved":0,"allocated":364462080,"data_stored":364263064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493888,"kb_used_data":255336,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93877952,"statfs":{"total":96636764160,"available":96131022848,"internally_reserved":0,"allocated":261464064,"data_stored":261260384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704572,"kb_used_data":357804,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93667268,"statfs":{"total":96636764160,"available":95915282432,"internally_reserved":0,"allocated":366391296,"data_stored":366187616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367243264,"data_stored":367243264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261230592,"data_stored":261230592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364560384,"data_stored":364560384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367714304,"data_stored":367714304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:17.738 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:17.738 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'5624", 'reported_seq': 5678, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:15.395734+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:15.395734+0000', 'last_peered': '2024-09-10T20:23:15.395734+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:15.395734+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5624, 'log_dups_size': 0, 'ondisk_log_size': 5624, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46071808, 'num_objects': 5624, 'num_object_clones': 0, 'num_object_copies': 22496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5624, 'num_write_kb': 44992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3658}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'5579", 'reported_seq': 5665, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:12.517152+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:12.517152+0000', 'last_peered': '2024-09-10T20:23:12.517152+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:12.517152+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5579, 'log_dups_size': 0, 'ondisk_log_size': 5579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45677136, 'num_objects': 5576, 'num_object_clones': 0, 'num_object_copies': 22304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5578, 'num_write_kb': 44614, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3614}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "38'5692", 'reported_seq': 9795, 'reported_epoch': 38, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-10T20:23:13.631409+0000', 'last_change': '2024-09-10T20:23:13.591911+0000', 'last_active': '2024-09-10T20:23:13.631409+0000', 'last_peered': '2024-09-10T20:23:13.631409+0000', 'last_clean': '2024-09-10T20:23:13.631409+0000', 'last_became_active': '2024-09-10T20:23:13.161057+0000', 'last_became_peered': '2024-09-10T20:23:13.161057+0000', 'last_unstale': '2024-09-10T20:23:13.631409+0000', 'last_undegraded': '2024-09-10T20:23:13.631409+0000', 'last_fullsized': '2024-09-10T20:23:13.631409+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 99, 'log_size': 5692, 'log_dups_size': 0, 'ondisk_log_size': 5692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46593184, 'num_objects': 5688, 'num_object_clones': 0, 'num_object_copies': 22752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5690, 'num_write_kb': 45509, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2007, 'num_bytes_recovered': 16408576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'5568", 'reported_seq': 5632, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:15.343879+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:15.343879+0000', 'last_peered': '2024-09-10T20:23:15.343879+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:15.343879+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5568, 'log_dups_size': 0, 'ondisk_log_size': 5568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45585568, 'num_objects': 5565, 'num_object_clones': 0, 'num_object_copies': 22260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5566, 'num_write_kb': 44525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3585}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'5558", 'reported_seq': 5612, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:15.157786+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:15.157786+0000', 'last_peered': '2024-09-10T20:23:15.157786+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:15.157786+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5558, 'log_dups_size': 0, 'ondisk_log_size': 5558, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45531136, 'num_objects': 5558, 'num_object_clones': 0, 'num_object_copies': 22232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5558, 'num_write_kb': 44464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3620}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:17.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'5592", 'reported_seq': 5669, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:14.766429+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:14.766429+0000', 'last_peered': '2024-09-10T20:23:14.766429+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:14.766429+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5592, 'log_dups_size': 0, 'ondisk_log_size': 5592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45757600, 'num_objects': 5586, 'num_object_clones': 0, 'num_object_copies': 22344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5590, 'num_write_kb': 44700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3633}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'5655", 'reported_seq': 5731, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:15.013526+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:15.013526+0000', 'last_peered': '2024-09-10T20:23:15.013526+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:15.013526+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5655, 'log_dups_size': 0, 'ondisk_log_size': 5655, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46281888, 'num_objects': 5650, 'num_object_clones': 0, 'num_object_copies': 22600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5653, 'num_write_kb': 45212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3607}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'5553", 'reported_seq': 5688, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:12.609386+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:12.609386+0000', 'last_peered': '2024-09-10T20:23:12.609386+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:12.609386+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5553, 'log_dups_size': 0, 'ondisk_log_size': 5553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45470880, 'num_objects': 5551, 'num_object_clones': 0, 'num_object_copies': 22204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1880, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5551, 'num_write_kb': 44406, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 188416, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3671}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'5563", 'reported_seq': 5630, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:15.378481+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:15.378481+0000', 'last_peered': '2024-09-10T20:23:15.378481+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:15.378481+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5563, 'log_dups_size': 0, 'ondisk_log_size': 5563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45526768, 'num_objects': 5558, 'num_object_clones': 0, 'num_object_copies': 22232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5560, 'num_write_kb': 44468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3602}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'5587", 'reported_seq': 5642, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:15.375263+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:15.375263+0000', 'last_peered': '2024-09-10T20:23:15.375263+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:15.375263+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5587, 'log_dups_size': 0, 'ondisk_log_size': 5587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45759056, 'num_objects': 5586, 'num_object_clones': 0, 'num_object_copies': 22344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5586, 'num_write_kb': 44687, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:17.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'5441", 'reported_seq': 5526, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:12.610522+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:12.610522+0000', 'last_peered': '2024-09-10T20:23:12.610522+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:12.610522+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5441, 'log_dups_size': 0, 'ondisk_log_size': 5441, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44563024, 'num_objects': 5440, 'num_object_clones': 0, 'num_object_copies': 21760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5440, 'num_write_kb': 43519, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:17.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:17.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'5583", 'reported_seq': 5646, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:15.160043+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:15.160043+0000', 'last_peered': '2024-09-10T20:23:15.160043+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:15.160043+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5583, 'log_dups_size': 0, 'ondisk_log_size': 5583, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45700256, 'num_objects': 5579, 'num_object_clones': 0, 'num_object_copies': 22316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5581, 'num_write_kb': 44644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:17.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:17.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'5718", 'reported_seq': 5807, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:12.479108+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:12.479108+0000', 'last_peered': '2024-09-10T20:23:12.479108+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:12.479108+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5718, 'log_dups_size': 0, 'ondisk_log_size': 5718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46814368, 'num_objects': 5715, 'num_object_clones': 0, 'num_object_copies': 22860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5716, 'num_write_kb': 45725, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3753}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:17.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:17.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'5593", 'reported_seq': 5669, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:14.980966+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:14.980966+0000', 'last_peered': '2024-09-10T20:23:14.980966+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:14.980966+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5593, 'log_dups_size': 0, 'ondisk_log_size': 5593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45773984, 'num_objects': 5588, 'num_object_clones': 0, 'num_object_copies': 22352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5591, 'num_write_kb': 44716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3609}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:17.741 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:17.799 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 216 20 90431 90411 3.26962 2.03906 0.0496606 0.0477766 2024-09-10T20:23:18.519 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:18.519 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25483/359360 objects degraded (7.091%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":287,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89840,"data_bytes":735935792,"bytes_used":2661990400,"bytes_avail":383885066240,"bytes_total":386547056640,"degraded_objects":25483,"degraded_total":359360,"degraded_ratio":0.07091217720391807,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":184234,"recovering_keys_per_sec":0,"num_objects_recovered":180,"num_bytes_recovered":1474560,"num_keys_recovered":0,"write_bytes_sec":2846428,"read_op_per_sec":0,"write_op_per_sec":347},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:18.541 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:18.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:18.801 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 217 20 90684 90664 3.26364 1.97656 0.0603604 0.0478619 2024-09-10T20:23:19.162 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:19.162 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:19.187 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-10T20:23:17.951663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":735935792,"num_objects":89840,"num_object_clones":0,"num_object_copies":359360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82871,"num_write_kb":662877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":17047552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89899,"ondisk_log_size":89899,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2599600,"kb_used_data":1333604,"kb_used_omap":14,"kb_used_meta":1265905,"kb_avail":374887760,"statfs":{"total":386547056640,"available":383885066240,"internally_reserved":0,"allocated":1365610496,"data_stored":1364795776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1296287286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,10,10,11,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22781952,"num_objects":2781,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2781,"num_write_kb":22248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003697"},"pg_stats":[{"pgid":"1.f","version":"38'5624","reported_seq":5678,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.395734+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:15.395734+0000","last_peered":"2024-09-10T20:23:15.395734+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:15.395734+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":44992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5665","reported_seq":5791,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:13.595664+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:13.595664+0000","last_peered":"2024-09-10T20:23:13.595664+0000","last_clean":"2024-09-10T20:23:13.595664+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:13.595664+0000","last_undegraded":"2024-09-10T20:23:13.595664+0000","last_fullsized":"2024-09-10T20:23:13.595664+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46358528,"num_objects":5659,"num_object_clones":0,"num_object_copies":22636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5658","reported_seq":5746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.623496+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:17.623496+0000","last_peered":"2024-09-10T20:23:17.623496+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:17.623496+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324304,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5657,"num_write_kb":45246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5692","reported_seq":9795,"reported_epoch":38,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:23:13.631409+0000","last_change":"2024-09-10T20:23:13.591911+0000","last_active":"2024-09-10T20:23:13.631409+0000","last_peered":"2024-09-10T20:23:13.631409+0000","last_clean":"2024-09-10T20:23:13.631409+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:13.631409+0000","last_undegraded":"2024-09-10T20:23:13.631409+0000","last_fullsized":"2024-09-10T20:23:13.631409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":99,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46593184,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5690,"num_write_kb":45509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2007,"num_bytes_recovered":16408576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5568","reported_seq":5632,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.343879+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:15.343879+0000","last_peered":"2024-09-10T20:23:15.343879+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:15.343879+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45585568,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5566,"num_write_kb":44525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5558","reported_seq":5612,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.157786+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:15.157786+0000","last_peered":"2024-09-10T20:23:15.157786+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:15.157786+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45531136,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":44464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5592","reported_seq":5669,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:14.766429+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:14.766429+0000","last_peered":"2024-09-10T20:23:14.766429+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:14.766429+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45757600,"num_objects":5586,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":44700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3633}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5655","reported_seq":5731,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.013526+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:15.013526+0000","last_peered":"2024-09-10T20:23:15.013526+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:15.013526+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46281888,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":45212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3607}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5592","reported_seq":9558,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:15.313824+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:15.313824+0000","last_peered":"2024-09-10T20:23:15.313824+0000","last_clean":"2024-09-10T20:23:15.313824+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:15.313824+0000","last_undegraded":"2024-09-10T20:23:15.313824+0000","last_fullsized":"2024-09-10T20:23:15.313824+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45718096,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":17160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5653","reported_seq":5899,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.673672+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:17.673672+0000","last_peered":"2024-09-10T20:23:17.673672+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:17.673672+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290080,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":45206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5563","reported_seq":5630,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.378481+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:15.378481+0000","last_peered":"2024-09-10T20:23:15.378481+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:15.378481+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45526768,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5587","reported_seq":5642,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.375263+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:15.375263+0000","last_peered":"2024-09-10T20:23:15.375263+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:15.375263+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45759056,"num_objects":5586,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":44687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5520","reported_seq":5607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.532349+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:17.532349+0000","last_peered":"2024-09-10T20:23:17.532349+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:17.532349+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45210192,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":44151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3571}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5583","reported_seq":5646,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:15.160043+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:15.160043+0000","last_peered":"2024-09-10T20:23:15.160043+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:15.160043+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700256,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3604}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'5796","reported_seq":5887,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.433823+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:17.433823+0000","last_peered":"2024-09-10T20:23:17.433823+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:17.433823+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5796,"log_dups_size":0,"ondisk_log_size":5796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47453344,"num_objects":5793,"num_object_clones":0,"num_object_copies":23172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5794,"num_write_kb":46349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3831}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5593","reported_seq":5669,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:14.980966+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:14.980966+0000","last_peered":"2024-09-10T20:23:14.980966+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:14.980966+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5593,"log_dups_size":0,"ondisk_log_size":5593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45773984,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5591,"num_write_kb":44716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":735935792,"num_objects":89840,"num_object_clones":0,"num_object_copies":359360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82871,"num_write_kb":662877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":17047552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1366511616,"data_stored":1366511616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89899,"ondisk_log_size":89899,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705896,"kb_used_data":359060,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93665944,"statfs":{"total":96636764160,"available":95913926656,"internally_reserved":0,"allocated":367677440,"data_stored":367473760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,5,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695244,"kb_used_data":361404,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93676596,"statfs":{"total":96636764160,"available":95924834304,"internally_reserved":0,"allocated":370077696,"data_stored":369874016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493888,"kb_used_data":255336,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93877952,"statfs":{"total":96636764160,"available":96131022848,"internally_reserved":0,"allocated":261464064,"data_stored":261260384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704572,"kb_used_data":357804,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93667268,"statfs":{"total":96636764160,"available":95915282432,"internally_reserved":0,"allocated":366391296,"data_stored":366187616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367243264,"data_stored":367243264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261230592,"data_stored":261230592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370323456,"data_stored":370323456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367714304,"data_stored":367714304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:19.804 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 218 20 90977 90957 3.25917 2.28906 0.0779457 0.047927 2024-09-10T20:23:20.801 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 219 20 91292 91272 3.25552 2.46094 0.0621719 0.0479846 2024-09-10T20:23:21.801 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:23:21.798704+0000 min lat: 0.00230373 max lat: 12.2408 avg lat: 0.0480369 2024-09-10T20:23:21.802 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:23:21.802 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 220 20 91608 91588 3.25194 2.46875 0.0488768 0.0480369 2024-09-10T20:23:22.189 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:22.775 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:22.775 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:22.796 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-10T20:23:21.952953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":744840496,"num_objects":90927,"num_object_clones":0,"num_object_copies":363708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83958,"num_write_kb":671573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":17047552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90986,"ondisk_log_size":90986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2579184,"kb_used_data":1350640,"kb_used_omap":14,"kb_used_meta":1228465,"kb_avail":374908176,"statfs":{"total":386547056640,"available":383905972224,"internally_reserved":0,"allocated":1383055360,"data_stored":1382240640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1257948726},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,11,15,4,17,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22953984,"num_objects":2802,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":22416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.851374"},"pg_stats":[{"pgid":"1.f","version":"38'5706","reported_seq":5759,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.551794+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:20.551794+0000","last_peered":"2024-09-10T20:23:20.551794+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:20.551794+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46743552,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5743","reported_seq":5869,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:18.428396+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:18.428396+0000","last_peered":"2024-09-10T20:23:18.428396+0000","last_clean":"2024-09-10T20:23:18.428396+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:18.428396+0000","last_undegraded":"2024-09-10T20:23:18.428396+0000","last_fullsized":"2024-09-10T20:23:18.428396+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46997504,"num_objects":5737,"num_object_clones":0,"num_object_copies":22948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":17472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5658","reported_seq":5746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.623496+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:17.623496+0000","last_peered":"2024-09-10T20:23:17.623496+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:17.623496+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324304,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5657,"num_write_kb":45246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5790","reported_seq":9948,"reported_epoch":38,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:23:18.662770+0000","last_change":"2024-09-10T20:23:13.591911+0000","last_active":"2024-09-10T20:23:18.662770+0000","last_peered":"2024-09-10T20:23:18.662770+0000","last_clean":"2024-09-10T20:23:18.662770+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:18.662770+0000","last_undegraded":"2024-09-10T20:23:18.662770+0000","last_fullsized":"2024-09-10T20:23:18.662770+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":5549,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396000,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2007,"num_bytes_recovered":16408576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5661","reported_seq":5725,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.483176+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:20.483176+0000","last_peered":"2024-09-10T20:23:20.483176+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:20.483176+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46347424,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5659,"num_write_kb":45269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5639","reported_seq":5693,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.451910+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:20.451910+0000","last_peered":"2024-09-10T20:23:20.451910+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:20.451910+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46194688,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5667","reported_seq":5744,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.047510+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:20.047510+0000","last_peered":"2024-09-10T20:23:20.047510+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:20.047510+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46372000,"num_objects":5661,"num_object_clones":0,"num_object_copies":22644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3708}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5761","reported_seq":5837,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:19.955356+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:19.955356+0000","last_peered":"2024-09-10T20:23:19.955356+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:19.955356+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150240,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":46060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5673","reported_seq":9637,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:20.493837+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:20.493837+0000","last_peered":"2024-09-10T20:23:20.493837+0000","last_clean":"2024-09-10T20:23:20.493837+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:20.493837+0000","last_undegraded":"2024-09-10T20:23:20.493837+0000","last_fullsized":"2024-09-10T20:23:20.493837+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46381648,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":17808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5653","reported_seq":5899,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.673672+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:17.673672+0000","last_peered":"2024-09-10T20:23:17.673672+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:17.673672+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290080,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":45206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5675","reported_seq":5741,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.506828+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:20.506828+0000","last_peered":"2024-09-10T20:23:20.506828+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:20.506828+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46444272,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5672,"num_write_kb":45364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5687","reported_seq":5743,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.493128+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:20.493128+0000","last_peered":"2024-09-10T20:23:20.493128+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:20.493128+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46578256,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5520","reported_seq":5607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.532349+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:17.532349+0000","last_peered":"2024-09-10T20:23:17.532349+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:17.532349+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45210192,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":44151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3571}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5686","reported_seq":5749,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.371356+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:20.371356+0000","last_peered":"2024-09-10T20:23:20.371356+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:20.371356+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5686,"log_dups_size":0,"ondisk_log_size":5686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544032,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":45468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'5796","reported_seq":5887,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.433823+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:17.433823+0000","last_peered":"2024-09-10T20:23:17.433823+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:17.433823+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5796,"log_dups_size":0,"ondisk_log_size":5796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47453344,"num_objects":5793,"num_object_clones":0,"num_object_copies":23172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5794,"num_write_kb":46349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3831}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5671","reported_seq":5746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.084543+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:20.084543+0000","last_peered":"2024-09-10T20:23:20.084543+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:20.084543+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46412960,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":744840496,"num_objects":90927,"num_object_clones":0,"num_object_copies":363708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83958,"num_write_kb":671573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":17047552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384173568,"data_stored":1384173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90986,"ondisk_log_size":90986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699088,"kb_used_data":364736,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93672752,"statfs":{"total":96636764160,"available":95920898048,"internally_reserved":0,"allocated":373489664,"data_stored":373285984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,0,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695244,"kb_used_data":361404,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93676596,"statfs":{"total":96636764160,"available":95924834304,"internally_reserved":0,"allocated":370077696,"data_stored":369874016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487184,"kb_used_data":261120,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93884656,"statfs":{"total":96636764160,"available":96137887744,"internally_reserved":0,"allocated":267386880,"data_stored":267183200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231468754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,2,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697668,"kb_used_data":363380,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93674172,"statfs":{"total":96636764160,"available":95922352128,"internally_reserved":0,"allocated":372101120,"data_stored":371897440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373055488,"data_stored":373055488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267173888,"data_stored":267173888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370323456,"data_stored":370323456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373620736,"data_stored":373620736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:22.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:22.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 221 20 91899 91879 3.24752 2.27344 0.0581473 0.048102 2024-09-10T20:23:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:23.396 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:23.422 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-10T20:23:21.952953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":744840496,"num_objects":90927,"num_object_clones":0,"num_object_copies":363708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83958,"num_write_kb":671573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":17047552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90986,"ondisk_log_size":90986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2579184,"kb_used_data":1350640,"kb_used_omap":14,"kb_used_meta":1228465,"kb_avail":374908176,"statfs":{"total":386547056640,"available":383905972224,"internally_reserved":0,"allocated":1383055360,"data_stored":1382240640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1257948726},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,11,15,4,17,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22953984,"num_objects":2802,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":22416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.851374"},"pg_stats":[{"pgid":"1.f","version":"38'5706","reported_seq":5759,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.551794+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:20.551794+0000","last_peered":"2024-09-10T20:23:20.551794+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:20.551794+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46743552,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5743","reported_seq":5869,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:18.428396+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:18.428396+0000","last_peered":"2024-09-10T20:23:18.428396+0000","last_clean":"2024-09-10T20:23:18.428396+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:18.428396+0000","last_undegraded":"2024-09-10T20:23:18.428396+0000","last_fullsized":"2024-09-10T20:23:18.428396+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46997504,"num_objects":5737,"num_object_clones":0,"num_object_copies":22948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":17472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5658","reported_seq":5746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.623496+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:17.623496+0000","last_peered":"2024-09-10T20:23:17.623496+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:17.623496+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324304,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5657,"num_write_kb":45246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5790","reported_seq":9948,"reported_epoch":38,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:23:18.662770+0000","last_change":"2024-09-10T20:23:13.591911+0000","last_active":"2024-09-10T20:23:18.662770+0000","last_peered":"2024-09-10T20:23:18.662770+0000","last_clean":"2024-09-10T20:23:18.662770+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:18.662770+0000","last_undegraded":"2024-09-10T20:23:18.662770+0000","last_fullsized":"2024-09-10T20:23:18.662770+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":5549,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396000,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2007,"num_bytes_recovered":16408576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5661","reported_seq":5725,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.483176+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:20.483176+0000","last_peered":"2024-09-10T20:23:20.483176+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:20.483176+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46347424,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5659,"num_write_kb":45269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5639","reported_seq":5693,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.451910+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:20.451910+0000","last_peered":"2024-09-10T20:23:20.451910+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:20.451910+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46194688,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5667","reported_seq":5744,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.047510+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:20.047510+0000","last_peered":"2024-09-10T20:23:20.047510+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:20.047510+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46372000,"num_objects":5661,"num_object_clones":0,"num_object_copies":22644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3708}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5761","reported_seq":5837,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:19.955356+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:19.955356+0000","last_peered":"2024-09-10T20:23:19.955356+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:19.955356+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150240,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":46060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5673","reported_seq":9637,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:20.493837+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:20.493837+0000","last_peered":"2024-09-10T20:23:20.493837+0000","last_clean":"2024-09-10T20:23:20.493837+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:20.493837+0000","last_undegraded":"2024-09-10T20:23:20.493837+0000","last_fullsized":"2024-09-10T20:23:20.493837+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46381648,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":17808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5653","reported_seq":5899,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.673672+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:17.673672+0000","last_peered":"2024-09-10T20:23:17.673672+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:17.673672+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290080,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":45206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5675","reported_seq":5741,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.506828+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:20.506828+0000","last_peered":"2024-09-10T20:23:20.506828+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:20.506828+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46444272,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5672,"num_write_kb":45364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5687","reported_seq":5743,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.493128+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:20.493128+0000","last_peered":"2024-09-10T20:23:20.493128+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:20.493128+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46578256,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5520","reported_seq":5607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.532349+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:17.532349+0000","last_peered":"2024-09-10T20:23:17.532349+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:17.532349+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45210192,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":44151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3571}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5686","reported_seq":5749,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.371356+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:20.371356+0000","last_peered":"2024-09-10T20:23:20.371356+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:20.371356+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5686,"log_dups_size":0,"ondisk_log_size":5686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544032,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":45468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'5796","reported_seq":5887,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:17.433823+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:17.433823+0000","last_peered":"2024-09-10T20:23:17.433823+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:17.433823+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5796,"log_dups_size":0,"ondisk_log_size":5796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47453344,"num_objects":5793,"num_object_clones":0,"num_object_copies":23172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5794,"num_write_kb":46349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3831}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5671","reported_seq":5746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.084543+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:20.084543+0000","last_peered":"2024-09-10T20:23:20.084543+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:20.084543+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46412960,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":744840496,"num_objects":90927,"num_object_clones":0,"num_object_copies":363708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83958,"num_write_kb":671573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":17047552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384173568,"data_stored":1384173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90986,"ondisk_log_size":90986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699088,"kb_used_data":364736,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93672752,"statfs":{"total":96636764160,"available":95920898048,"internally_reserved":0,"allocated":373489664,"data_stored":373285984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,0,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695244,"kb_used_data":361404,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93676596,"statfs":{"total":96636764160,"available":95924834304,"internally_reserved":0,"allocated":370077696,"data_stored":369874016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487184,"kb_used_data":261120,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93884656,"statfs":{"total":96636764160,"available":96137887744,"internally_reserved":0,"allocated":267386880,"data_stored":267183200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231468754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,2,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697668,"kb_used_data":363380,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93674172,"statfs":{"total":96636764160,"available":95922352128,"internally_reserved":0,"allocated":372101120,"data_stored":371897440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373055488,"data_stored":373055488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267173888,"data_stored":267173888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370323456,"data_stored":370323456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373620736,"data_stored":373620736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:23.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:23.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'5706", 'reported_seq': 5759, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.551794+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:20.551794+0000', 'last_peered': '2024-09-10T20:23:20.551794+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:20.551794+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5706, 'log_dups_size': 0, 'ondisk_log_size': 5706, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46743552, 'num_objects': 5706, 'num_object_clones': 0, 'num_object_copies': 22824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5706, 'num_write_kb': 45648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:23.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:23.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'5658", 'reported_seq': 5746, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:17.623496+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:17.623496+0000', 'last_peered': '2024-09-10T20:23:17.623496+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:17.623496+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5658, 'log_dups_size': 0, 'ondisk_log_size': 5658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46324304, 'num_objects': 5655, 'num_object_clones': 0, 'num_object_copies': 22620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5657, 'num_write_kb': 45246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3693}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:23.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "38'5790", 'reported_seq': 9948, 'reported_epoch': 38, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-10T20:23:18.662770+0000', 'last_change': '2024-09-10T20:23:13.591911+0000', 'last_active': '2024-09-10T20:23:18.662770+0000', 'last_peered': '2024-09-10T20:23:18.662770+0000', 'last_clean': '2024-09-10T20:23:18.662770+0000', 'last_became_active': '2024-09-10T20:23:13.161057+0000', 'last_became_peered': '2024-09-10T20:23:13.161057+0000', 'last_unstale': '2024-09-10T20:23:18.662770+0000', 'last_undegraded': '2024-09-10T20:23:18.662770+0000', 'last_fullsized': '2024-09-10T20:23:18.662770+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 5549, 'log_size': 5790, 'log_dups_size': 0, 'ondisk_log_size': 5790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 5s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47396000, 'num_objects': 5786, 'num_object_clones': 0, 'num_object_copies': 23144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5788, 'num_write_kb': 46293, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2007, 'num_bytes_recovered': 16408576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'5661", 'reported_seq': 5725, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.483176+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:20.483176+0000', 'last_peered': '2024-09-10T20:23:20.483176+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:20.483176+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5661, 'log_dups_size': 0, 'ondisk_log_size': 5661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46347424, 'num_objects': 5658, 'num_object_clones': 0, 'num_object_copies': 22632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5659, 'num_write_kb': 45269, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3678}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'5639", 'reported_seq': 5693, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.451910+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:20.451910+0000', 'last_peered': '2024-09-10T20:23:20.451910+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:20.451910+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5639, 'log_dups_size': 0, 'ondisk_log_size': 5639, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46194688, 'num_objects': 5639, 'num_object_clones': 0, 'num_object_copies': 22556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5639, 'num_write_kb': 45112, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3701}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'5667", 'reported_seq': 5744, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.047510+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:20.047510+0000', 'last_peered': '2024-09-10T20:23:20.047510+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:20.047510+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5667, 'log_dups_size': 0, 'ondisk_log_size': 5667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46372000, 'num_objects': 5661, 'num_object_clones': 0, 'num_object_copies': 22644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5665, 'num_write_kb': 45300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3708}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'5761", 'reported_seq': 5837, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:19.955356+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:19.955356+0000', 'last_peered': '2024-09-10T20:23:19.955356+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:19.955356+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5761, 'log_dups_size': 0, 'ondisk_log_size': 5761, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47150240, 'num_objects': 5756, 'num_object_clones': 0, 'num_object_copies': 23024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5759, 'num_write_kb': 46060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3713}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:23.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'5653", 'reported_seq': 5899, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:17.673672+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:17.673672+0000', 'last_peered': '2024-09-10T20:23:17.673672+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:17.673672+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5653, 'log_dups_size': 0, 'ondisk_log_size': 5653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46290080, 'num_objects': 5651, 'num_object_clones': 0, 'num_object_copies': 22604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1825, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5651, 'num_write_kb': 45206, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 80, 'num_bytes_recovered': 638976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3826}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'5675", 'reported_seq': 5741, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.506828+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:20.506828+0000', 'last_peered': '2024-09-10T20:23:20.506828+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:20.506828+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5675, 'log_dups_size': 0, 'ondisk_log_size': 5675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46444272, 'num_objects': 5670, 'num_object_clones': 0, 'num_object_copies': 22680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5672, 'num_write_kb': 45364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3714}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'5687", 'reported_seq': 5743, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.493128+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:20.493128+0000', 'last_peered': '2024-09-10T20:23:20.493128+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:20.493128+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5687, 'log_dups_size': 0, 'ondisk_log_size': 5687, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46578256, 'num_objects': 5686, 'num_object_clones': 0, 'num_object_copies': 22744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5686, 'num_write_kb': 45487, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3708}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'5520", 'reported_seq': 5607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:17.532349+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:17.532349+0000', 'last_peered': '2024-09-10T20:23:17.532349+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:17.532349+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5520, 'log_dups_size': 0, 'ondisk_log_size': 5520, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45210192, 'num_objects': 5519, 'num_object_clones': 0, 'num_object_copies': 22076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5519, 'num_write_kb': 44151, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3571}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'5686", 'reported_seq': 5749, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.371356+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:20.371356+0000', 'last_peered': '2024-09-10T20:23:20.371356+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:20.371356+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5686, 'log_dups_size': 0, 'ondisk_log_size': 5686, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46544032, 'num_objects': 5682, 'num_object_clones': 0, 'num_object_copies': 22728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5684, 'num_write_kb': 45468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3707}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:23.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:23.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'5796", 'reported_seq': 5887, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:17.433823+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:17.433823+0000', 'last_peered': '2024-09-10T20:23:17.433823+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:17.433823+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5796, 'log_dups_size': 0, 'ondisk_log_size': 5796, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47453344, 'num_objects': 5793, 'num_object_clones': 0, 'num_object_copies': 23172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5794, 'num_write_kb': 46349, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3831}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:23.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:23.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'5671", 'reported_seq': 5746, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:20.084543+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:20.084543+0000', 'last_peered': '2024-09-10T20:23:20.084543+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:20.084543+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5671, 'log_dups_size': 0, 'ondisk_log_size': 5671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46412960, 'num_objects': 5666, 'num_object_clones': 0, 'num_object_copies': 22664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5669, 'num_write_kb': 45340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:23.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:23.801 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 222 20 92174 92154 3.24256 2.14844 0.109194 0.0481747 2024-09-10T20:23:24.147 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:24.147 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25483/363708 objects degraded (7.006%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":90927,"data_bytes":744840496,"bytes_used":2641084416,"bytes_avail":383905972224,"bytes_total":386547056640,"degraded_objects":25483,"degraded_total":363708,"degraded_ratio":0.070064447303881131,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":79829,"recovering_keys_per_sec":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"write_bytes_sec":2330028,"read_op_per_sec":0,"write_op_per_sec":284},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:24.180 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:24.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:24.791 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:24.792 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:24.801 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 223 20 92445 92425 3.23752 2.11719 0.0563561 0.0482509 2024-09-10T20:23:24.818 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-10T20:23:23.953674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749430928,"num_objects":91487,"num_object_clones":0,"num_object_copies":365948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78782,"num_write_kb":630176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91548,"ondisk_log_size":91548,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2599408,"kb_used_data":1362544,"kb_used_omap":14,"kb_used_meta":1236785,"kb_avail":374887952,"statfs":{"total":386547056640,"available":383885262848,"internally_reserved":0,"allocated":1395245056,"data_stored":1394430336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1266468406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,14,15,7,20,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24431456,"num_objects":2982,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2754,"num_write_kb":-22021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1887,"num_bytes_recovered":-15425536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.843252"},"pg_stats":[{"pgid":"1.f","version":"38'5706","reported_seq":5759,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.551794+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:20.551794+0000","last_peered":"2024-09-10T20:23:20.551794+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:20.551794+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46743552,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5838","reported_seq":5964,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:23.788866+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:23.788866+0000","last_peered":"2024-09-10T20:23:23.788866+0000","last_clean":"2024-09-10T20:23:23.788866+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:23.788866+0000","last_undegraded":"2024-09-10T20:23:23.788866+0000","last_fullsized":"2024-09-10T20:23:23.788866+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5838,"log_dups_size":0,"ondisk_log_size":5838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47775744,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":18232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5762","reported_seq":5850,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:22.642435+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:22.642435+0000","last_peered":"2024-09-10T20:23:22.642435+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:22.642435+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47176272,"num_objects":5759,"num_object_clones":0,"num_object_copies":23036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":46078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5892","reported_seq":10056,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:23.630101+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:23.630101+0000","last_peered":"2024-09-10T20:23:23.630101+0000","last_clean":"2024-09-10T20:23:23.630101+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:23.630101+0000","last_undegraded":"2024-09-10T20:23:23.630101+0000","last_fullsized":"2024-09-10T20:23:23.630101+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5661","reported_seq":5725,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.483176+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:20.483176+0000","last_peered":"2024-09-10T20:23:20.483176+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:20.483176+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46347424,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5659,"num_write_kb":45269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5639","reported_seq":5693,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.451910+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:20.451910+0000","last_peered":"2024-09-10T20:23:20.451910+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:20.451910+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46194688,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5667","reported_seq":5744,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.047510+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:20.047510+0000","last_peered":"2024-09-10T20:23:20.047510+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:20.047510+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46372000,"num_objects":5661,"num_object_clones":0,"num_object_copies":22644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3708}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5761","reported_seq":5837,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:19.955356+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:19.955356+0000","last_peered":"2024-09-10T20:23:19.955356+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:19.955356+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150240,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":46060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5673","reported_seq":9637,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:20.493837+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:20.493837+0000","last_peered":"2024-09-10T20:23:20.493837+0000","last_clean":"2024-09-10T20:23:20.493837+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:20.493837+0000","last_undegraded":"2024-09-10T20:23:20.493837+0000","last_fullsized":"2024-09-10T20:23:20.493837+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46381648,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":17808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5735","reported_seq":6075,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:22.847813+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:22.847813+0000","last_peered":"2024-09-10T20:23:22.847813+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:22.847813+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961824,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5733,"num_write_kb":45862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5675","reported_seq":5741,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.506828+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:20.506828+0000","last_peered":"2024-09-10T20:23:20.506828+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:20.506828+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46444272,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5672,"num_write_kb":45364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5687","reported_seq":5743,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.493128+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:20.493128+0000","last_peered":"2024-09-10T20:23:20.493128+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:20.493128+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46578256,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5606","reported_seq":5693,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:22.739459+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:22.739459+0000","last_peered":"2024-09-10T20:23:22.739459+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:22.739459+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914704,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5686","reported_seq":5749,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.371356+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:20.371356+0000","last_peered":"2024-09-10T20:23:20.371356+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:20.371356+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5686,"log_dups_size":0,"ondisk_log_size":5686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544032,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":45468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'5889","reported_seq":5980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:22.615203+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:22.615203+0000","last_peered":"2024-09-10T20:23:22.615203+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:22.615203+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48215200,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5887,"num_write_kb":47093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3924}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5671","reported_seq":5746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:20.084543+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:20.084543+0000","last_peered":"2024-09-10T20:23:20.084543+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:20.084543+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46412960,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":749430928,"num_objects":91487,"num_object_clones":0,"num_object_copies":365948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78782,"num_write_kb":630176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1396666368,"data_stored":1396666368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91548,"ondisk_log_size":91548,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699088,"kb_used_data":364736,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93672752,"statfs":{"total":96636764160,"available":95920898048,"internally_reserved":0,"allocated":373489664,"data_stored":373285984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,0,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705192,"kb_used_data":367192,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93666648,"statfs":{"total":96636764160,"available":95914647552,"internally_reserved":0,"allocated":376004608,"data_stored":375800928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,5,5,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497460,"kb_used_data":267236,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93874380,"statfs":{"total":96636764160,"available":96127365120,"internally_reserved":0,"allocated":273649664,"data_stored":273445984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,7,2,9,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697668,"kb_used_data":363380,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93674172,"statfs":{"total":96636764160,"available":95922352128,"internally_reserved":0,"allocated":372101120,"data_stored":371897440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373055488,"data_stored":373055488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273522688,"data_stored":273522688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376467456,"data_stored":376467456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373620736,"data_stored":373620736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:25.801 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 224 20 92751 92731 3.23374 2.39062 0.0539605 0.0483074 2024-09-10T20:23:26.802 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 225 20 93042 93022 3.22947 2.27344 0.136892 0.0483705 2024-09-10T20:23:27.802 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 226 20 93357 93337 3.22607 2.46094 0.0685027 0.0484227 2024-09-10T20:23:27.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:28.423 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:28.423 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:28.444 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-10T20:23:27.955266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":760105104,"num_objects":92790,"num_object_clones":0,"num_object_copies":371160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80085,"num_write_kb":640600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92851,"ondisk_log_size":92851,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2630632,"kb_used_data":1381284,"kb_used_omap":14,"kb_used_meta":1249265,"kb_avail":374856728,"statfs":{"total":386547056640,"available":383853289472,"internally_reserved":0,"allocated":1414434816,"data_stored":1413620096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1279247926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,7,3,8,9,21,18,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26921824,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2450,"num_write_kb":-19589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1860,"num_bytes_recovered":-15204352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005227"},"pg_stats":[{"pgid":"1.f","version":"38'5793","reported_seq":5847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.433045+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:25.433045+0000","last_peered":"2024-09-10T20:23:25.433045+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:25.433045+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5793,"log_dups_size":0,"ondisk_log_size":5793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47456256,"num_objects":5793,"num_object_clones":0,"num_object_copies":23172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":46344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5838","reported_seq":5964,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:23.788866+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:23.788866+0000","last_peered":"2024-09-10T20:23:23.788866+0000","last_clean":"2024-09-10T20:23:23.788866+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:23.788866+0000","last_undegraded":"2024-09-10T20:23:23.788866+0000","last_fullsized":"2024-09-10T20:23:23.788866+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5838,"log_dups_size":0,"ondisk_log_size":5838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47775744,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":18232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5846","reported_seq":5934,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.804428+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:27.804428+0000","last_peered":"2024-09-10T20:23:27.804428+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:27.804428+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47864400,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":46750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3881}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5892","reported_seq":10056,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:23.630101+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:23.630101+0000","last_peered":"2024-09-10T20:23:23.630101+0000","last_clean":"2024-09-10T20:23:23.630101+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:23.630101+0000","last_undegraded":"2024-09-10T20:23:23.630101+0000","last_fullsized":"2024-09-10T20:23:23.630101+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5780","reported_seq":5844,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.604826+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:25.604826+0000","last_peered":"2024-09-10T20:23:25.604826+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:25.604826+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47322272,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5778,"num_write_kb":46221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5742","reported_seq":5796,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.535606+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:25.535606+0000","last_peered":"2024-09-10T20:23:25.535606+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:25.535606+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47038464,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":45936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5745","reported_seq":5822,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.107472+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:25.107472+0000","last_peered":"2024-09-10T20:23:25.107472+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:25.107472+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47010976,"num_objects":5739,"num_object_clones":0,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5743,"num_write_kb":45924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5856","reported_seq":5932,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.070499+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:25.070499+0000","last_peered":"2024-09-10T20:23:25.070499+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:25.070499+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5856,"log_dups_size":0,"ondisk_log_size":5856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47928480,"num_objects":5851,"num_object_clones":0,"num_object_copies":23404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5854,"num_write_kb":46820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5758","reported_seq":9724,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:25.402240+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:25.402240+0000","last_peered":"2024-09-10T20:23:25.402240+0000","last_clean":"2024-09-10T20:23:25.402240+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:25.402240+0000","last_undegraded":"2024-09-10T20:23:25.402240+0000","last_fullsized":"2024-09-10T20:23:25.402240+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47077968,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2311,"num_write_kb":18488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5821","reported_seq":6252,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.850141+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:27.850141+0000","last_peered":"2024-09-10T20:23:27.850141+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:27.850141+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47666336,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":46550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5792","reported_seq":5859,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.394850+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:25.394850+0000","last_peered":"2024-09-10T20:23:25.394850+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:25.394850+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47402736,"num_objects":5787,"num_object_clones":0,"num_object_copies":23148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5789,"num_write_kb":46300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5773","reported_seq":5829,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.553108+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:25.553108+0000","last_peered":"2024-09-10T20:23:25.553108+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:25.553108+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47282768,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5772,"num_write_kb":46175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5703","reported_seq":5790,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.778140+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:27.778140+0000","last_peered":"2024-09-10T20:23:27.778140+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:27.778140+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46709328,"num_objects":5702,"num_object_clones":0,"num_object_copies":22808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":45615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5776","reported_seq":5839,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.597741+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:25.597741+0000","last_peered":"2024-09-10T20:23:25.597741+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:25.597741+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":46188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'5979","reported_seq":6070,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.776219+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:27.776219+0000","last_peered":"2024-09-10T20:23:27.776219+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:27.776219+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5979,"log_dups_size":0,"ondisk_log_size":5979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48952480,"num_objects":5976,"num_object_clones":0,"num_object_copies":23904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":47813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5757","reported_seq":5833,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.111343+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:25.111343+0000","last_peered":"2024-09-10T20:23:25.111343+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:25.111343+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5755,"num_write_kb":46028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":760105104,"num_objects":92790,"num_object_clones":0,"num_object_copies":371160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80085,"num_write_kb":640600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1414832128,"data_stored":1414832128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92851,"ondisk_log_size":92851,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709124,"kb_used_data":370608,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93662716,"statfs":{"total":96636764160,"available":95910621184,"internally_reserved":0,"allocated":379502592,"data_stored":379298912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715084,"kb_used_data":372924,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93656756,"statfs":{"total":96636764160,"available":95904518144,"internally_reserved":0,"allocated":381874176,"data_stored":381670496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,3,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497460,"kb_used_data":267236,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93874380,"statfs":{"total":96636764160,"available":96127365120,"internally_reserved":0,"allocated":273649664,"data_stored":273445984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,7,2,9,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708964,"kb_used_data":370516,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93662876,"statfs":{"total":96636764160,"available":95910785024,"internally_reserved":0,"allocated":379408384,"data_stored":379204704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379121664,"data_stored":379121664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273522688,"data_stored":273522688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382464000,"data_stored":382464000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379723776,"data_stored":379723776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:28.445 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:28.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 227 20 93612 93592 3.22063 1.99219 0.0703652 0.0485011 2024-09-10T20:23:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:29.000 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:29.028 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-10T20:23:27.955266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":760105104,"num_objects":92790,"num_object_clones":0,"num_object_copies":371160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80085,"num_write_kb":640600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92851,"ondisk_log_size":92851,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2630632,"kb_used_data":1381284,"kb_used_omap":14,"kb_used_meta":1249265,"kb_avail":374856728,"statfs":{"total":386547056640,"available":383853289472,"internally_reserved":0,"allocated":1414434816,"data_stored":1413620096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1279247926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,7,3,8,9,21,18,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26921824,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2450,"num_write_kb":-19589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1860,"num_bytes_recovered":-15204352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005227"},"pg_stats":[{"pgid":"1.f","version":"38'5793","reported_seq":5847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.433045+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:25.433045+0000","last_peered":"2024-09-10T20:23:25.433045+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:25.433045+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5793,"log_dups_size":0,"ondisk_log_size":5793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47456256,"num_objects":5793,"num_object_clones":0,"num_object_copies":23172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":46344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5838","reported_seq":5964,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:23.788866+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:23.788866+0000","last_peered":"2024-09-10T20:23:23.788866+0000","last_clean":"2024-09-10T20:23:23.788866+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:23.788866+0000","last_undegraded":"2024-09-10T20:23:23.788866+0000","last_fullsized":"2024-09-10T20:23:23.788866+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5838,"log_dups_size":0,"ondisk_log_size":5838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47775744,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":18232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5846","reported_seq":5934,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.804428+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:27.804428+0000","last_peered":"2024-09-10T20:23:27.804428+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:27.804428+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47864400,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":46750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3881}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5892","reported_seq":10056,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:23.630101+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:23.630101+0000","last_peered":"2024-09-10T20:23:23.630101+0000","last_clean":"2024-09-10T20:23:23.630101+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:23.630101+0000","last_undegraded":"2024-09-10T20:23:23.630101+0000","last_fullsized":"2024-09-10T20:23:23.630101+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5780","reported_seq":5844,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.604826+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:25.604826+0000","last_peered":"2024-09-10T20:23:25.604826+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:25.604826+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47322272,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5778,"num_write_kb":46221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5742","reported_seq":5796,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.535606+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:25.535606+0000","last_peered":"2024-09-10T20:23:25.535606+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:25.535606+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47038464,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":45936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5745","reported_seq":5822,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.107472+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:25.107472+0000","last_peered":"2024-09-10T20:23:25.107472+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:25.107472+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47010976,"num_objects":5739,"num_object_clones":0,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5743,"num_write_kb":45924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5856","reported_seq":5932,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.070499+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:25.070499+0000","last_peered":"2024-09-10T20:23:25.070499+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:25.070499+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5856,"log_dups_size":0,"ondisk_log_size":5856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47928480,"num_objects":5851,"num_object_clones":0,"num_object_copies":23404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5854,"num_write_kb":46820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5758","reported_seq":9724,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:25.402240+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:25.402240+0000","last_peered":"2024-09-10T20:23:25.402240+0000","last_clean":"2024-09-10T20:23:25.402240+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:25.402240+0000","last_undegraded":"2024-09-10T20:23:25.402240+0000","last_fullsized":"2024-09-10T20:23:25.402240+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47077968,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2311,"num_write_kb":18488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5821","reported_seq":6252,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.850141+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:27.850141+0000","last_peered":"2024-09-10T20:23:27.850141+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:27.850141+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47666336,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":46550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5792","reported_seq":5859,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.394850+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:25.394850+0000","last_peered":"2024-09-10T20:23:25.394850+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:25.394850+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47402736,"num_objects":5787,"num_object_clones":0,"num_object_copies":23148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5789,"num_write_kb":46300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5773","reported_seq":5829,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.553108+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:25.553108+0000","last_peered":"2024-09-10T20:23:25.553108+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:25.553108+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47282768,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5772,"num_write_kb":46175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5703","reported_seq":5790,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.778140+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:27.778140+0000","last_peered":"2024-09-10T20:23:27.778140+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:27.778140+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46709328,"num_objects":5702,"num_object_clones":0,"num_object_copies":22808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":45615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5776","reported_seq":5839,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.597741+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:25.597741+0000","last_peered":"2024-09-10T20:23:25.597741+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:25.597741+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":46188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'5979","reported_seq":6070,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.776219+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:27.776219+0000","last_peered":"2024-09-10T20:23:27.776219+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:27.776219+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5979,"log_dups_size":0,"ondisk_log_size":5979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48952480,"num_objects":5976,"num_object_clones":0,"num_object_copies":23904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":47813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5757","reported_seq":5833,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.111343+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:25.111343+0000","last_peered":"2024-09-10T20:23:25.111343+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:25.111343+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5755,"num_write_kb":46028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":760105104,"num_objects":92790,"num_object_clones":0,"num_object_copies":371160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80085,"num_write_kb":640600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1414832128,"data_stored":1414832128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92851,"ondisk_log_size":92851,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709124,"kb_used_data":370608,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93662716,"statfs":{"total":96636764160,"available":95910621184,"internally_reserved":0,"allocated":379502592,"data_stored":379298912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715084,"kb_used_data":372924,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93656756,"statfs":{"total":96636764160,"available":95904518144,"internally_reserved":0,"allocated":381874176,"data_stored":381670496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,3,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497460,"kb_used_data":267236,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93874380,"statfs":{"total":96636764160,"available":96127365120,"internally_reserved":0,"allocated":273649664,"data_stored":273445984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,7,2,9,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708964,"kb_used_data":370516,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93662876,"statfs":{"total":96636764160,"available":95910785024,"internally_reserved":0,"allocated":379408384,"data_stored":379204704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379121664,"data_stored":379121664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273522688,"data_stored":273522688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382464000,"data_stored":382464000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379723776,"data_stored":379723776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:29.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:29.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'5793", 'reported_seq': 5847, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.433045+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:25.433045+0000', 'last_peered': '2024-09-10T20:23:25.433045+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:25.433045+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5793, 'log_dups_size': 0, 'ondisk_log_size': 5793, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47456256, 'num_objects': 5793, 'num_object_clones': 0, 'num_object_copies': 23172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5793, 'num_write_kb': 46344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3827}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:29.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'5846", 'reported_seq': 5934, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:27.804428+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:27.804428+0000', 'last_peered': '2024-09-10T20:23:27.804428+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:27.804428+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5846, 'log_dups_size': 0, 'ondisk_log_size': 5846, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47864400, 'num_objects': 5843, 'num_object_clones': 0, 'num_object_copies': 23372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5845, 'num_write_kb': 46750, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3881}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'5780", 'reported_seq': 5844, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.604826+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:25.604826+0000', 'last_peered': '2024-09-10T20:23:25.604826+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:25.604826+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5780, 'log_dups_size': 0, 'ondisk_log_size': 5780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47322272, 'num_objects': 5777, 'num_object_clones': 0, 'num_object_copies': 23108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5778, 'num_write_kb': 46221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3797}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'5742", 'reported_seq': 5796, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.535606+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:25.535606+0000', 'last_peered': '2024-09-10T20:23:25.535606+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:25.535606+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5742, 'log_dups_size': 0, 'ondisk_log_size': 5742, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47038464, 'num_objects': 5742, 'num_object_clones': 0, 'num_object_copies': 22968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5742, 'num_write_kb': 45936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3804}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'5745", 'reported_seq': 5822, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.107472+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:25.107472+0000', 'last_peered': '2024-09-10T20:23:25.107472+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:25.107472+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5745, 'log_dups_size': 0, 'ondisk_log_size': 5745, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47010976, 'num_objects': 5739, 'num_object_clones': 0, 'num_object_copies': 22956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5743, 'num_write_kb': 45924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3786}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'5856", 'reported_seq': 5932, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.070499+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:25.070499+0000', 'last_peered': '2024-09-10T20:23:25.070499+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:25.070499+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5856, 'log_dups_size': 0, 'ondisk_log_size': 5856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47928480, 'num_objects': 5851, 'num_object_clones': 0, 'num_object_copies': 23404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5854, 'num_write_kb': 46820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3808}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'5821", 'reported_seq': 6252, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:27.850141+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:27.850141+0000', 'last_peered': '2024-09-10T20:23:27.850141+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:27.850141+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5821, 'log_dups_size': 0, 'ondisk_log_size': 5821, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47666336, 'num_objects': 5819, 'num_object_clones': 0, 'num_object_copies': 23276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1733, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5819, 'num_write_kb': 46550, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 172, 'num_bytes_recovered': 1392640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4086}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'5792", 'reported_seq': 5859, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.394850+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:25.394850+0000', 'last_peered': '2024-09-10T20:23:25.394850+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:25.394850+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5792, 'log_dups_size': 0, 'ondisk_log_size': 5792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47402736, 'num_objects': 5787, 'num_object_clones': 0, 'num_object_copies': 23148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5789, 'num_write_kb': 46300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3831}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'5773", 'reported_seq': 5829, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.553108+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:25.553108+0000', 'last_peered': '2024-09-10T20:23:25.553108+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:25.553108+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5773, 'log_dups_size': 0, 'ondisk_log_size': 5773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47282768, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 23088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5772, 'num_write_kb': 46175, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'5703", 'reported_seq': 5790, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:27.778140+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:27.778140+0000', 'last_peered': '2024-09-10T20:23:27.778140+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:27.778140+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5703, 'log_dups_size': 0, 'ondisk_log_size': 5703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46709328, 'num_objects': 5702, 'num_object_clones': 0, 'num_object_copies': 22808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5702, 'num_write_kb': 45615, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'5776", 'reported_seq': 5839, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.597741+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:25.597741+0000', 'last_peered': '2024-09-10T20:23:25.597741+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:25.597741+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5776, 'log_dups_size': 0, 'ondisk_log_size': 5776, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47281312, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 23088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5774, 'num_write_kb': 46188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'5979", 'reported_seq': 6070, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:27.776219+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:27.776219+0000', 'last_peered': '2024-09-10T20:23:27.776219+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:27.776219+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5979, 'log_dups_size': 0, 'ondisk_log_size': 5979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48952480, 'num_objects': 5976, 'num_object_clones': 0, 'num_object_copies': 23904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5977, 'num_write_kb': 47813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:29.031 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:29.032 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'5757", 'reported_seq': 5833, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:25.111343+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:25.111343+0000', 'last_peered': '2024-09-10T20:23:25.111343+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:25.111343+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5757, 'log_dups_size': 0, 'ondisk_log_size': 5757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47117472, 'num_objects': 5752, 'num_object_clones': 0, 'num_object_copies': 23008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5755, 'num_write_kb': 46028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:29.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:29.759 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:29.760 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25391/371160 objects degraded (6.841%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":298,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92790,"data_bytes":760105104,"bytes_used":2693767168,"bytes_avail":383853289472,"bytes_total":386547056640,"degraded_objects":25391,"degraded_total":371160,"degraded_ratio":0.068409850199374928},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:29.792 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:23:29.793 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:29.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 228 20 93889 93869 3.21598 2.16406 0.0564432 0.0485732 2024-09-10T20:23:30.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 229 20 94166 94146 3.21139 2.16406 0.0542809 0.0486434 2024-09-10T20:23:31.004 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:31.005 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:31.025 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-10T20:23:29.956008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":761522320,"num_objects":92963,"num_object_clones":0,"num_object_copies":371852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80258,"num_write_kb":641984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93024,"ondisk_log_size":93024,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2640672,"kb_used_data":1387164,"kb_used_omap":14,"kb_used_meta":1253425,"kb_avail":374846688,"statfs":{"total":386547056640,"available":383843008512,"internally_reserved":0,"allocated":1420455936,"data_stored":1419641216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1283507766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,6,3,9,16,19,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25586528,"num_objects":3123,"num_object_clones":0,"num_object_copies":12492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2613,"num_write_kb":-20893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1915,"num_bytes_recovered":-15654912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004344"},"pg_stats":[{"pgid":"1.f","version":"38'5793","reported_seq":5847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.433045+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:25.433045+0000","last_peered":"2024-09-10T20:23:25.433045+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:25.433045+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5793,"log_dups_size":0,"ondisk_log_size":5793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47456256,"num_objects":5793,"num_object_clones":0,"num_object_copies":23172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":46344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5918","reported_seq":6044,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:28.650724+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:28.650724+0000","last_peered":"2024-09-10T20:23:28.650724+0000","last_clean":"2024-09-10T20:23:28.650724+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:28.650724+0000","last_undegraded":"2024-09-10T20:23:28.650724+0000","last_fullsized":"2024-09-10T20:23:28.650724+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2359,"num_write_kb":18872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5846","reported_seq":5934,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.804428+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:27.804428+0000","last_peered":"2024-09-10T20:23:27.804428+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:27.804428+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47864400,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":46750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3881}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'5985","reported_seq":10149,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:28.685932+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:28.685932+0000","last_peered":"2024-09-10T20:23:28.685932+0000","last_clean":"2024-09-10T20:23:28.685932+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:28.685932+0000","last_undegraded":"2024-09-10T20:23:28.685932+0000","last_fullsized":"2024-09-10T20:23:28.685932+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":5985,"log_dups_size":0,"ondisk_log_size":5985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48979968,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":245,"num_write_kb":1960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5780","reported_seq":5844,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.604826+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:25.604826+0000","last_peered":"2024-09-10T20:23:25.604826+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:25.604826+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47322272,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5778,"num_write_kb":46221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5742","reported_seq":5796,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.535606+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:25.535606+0000","last_peered":"2024-09-10T20:23:25.535606+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:25.535606+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47038464,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":45936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5745","reported_seq":5822,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.107472+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:25.107472+0000","last_peered":"2024-09-10T20:23:25.107472+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:25.107472+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47010976,"num_objects":5739,"num_object_clones":0,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5743,"num_write_kb":45924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5856","reported_seq":5932,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.070499+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:25.070499+0000","last_peered":"2024-09-10T20:23:25.070499+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:25.070499+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5856,"log_dups_size":0,"ondisk_log_size":5856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47928480,"num_objects":5851,"num_object_clones":0,"num_object_copies":23404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5854,"num_write_kb":46820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5758","reported_seq":9724,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:25.402240+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:25.402240+0000","last_peered":"2024-09-10T20:23:25.402240+0000","last_clean":"2024-09-10T20:23:25.402240+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:25.402240+0000","last_undegraded":"2024-09-10T20:23:25.402240+0000","last_fullsized":"2024-09-10T20:23:25.402240+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47077968,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2311,"num_write_kb":18488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5821","reported_seq":6252,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.850141+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:27.850141+0000","last_peered":"2024-09-10T20:23:27.850141+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:27.850141+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47666336,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":46550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5792","reported_seq":5859,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.394850+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:25.394850+0000","last_peered":"2024-09-10T20:23:25.394850+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:25.394850+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47402736,"num_objects":5787,"num_object_clones":0,"num_object_copies":23148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5789,"num_write_kb":46300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5773","reported_seq":5829,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.553108+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:25.553108+0000","last_peered":"2024-09-10T20:23:25.553108+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:25.553108+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47282768,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5772,"num_write_kb":46175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5703","reported_seq":5790,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.778140+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:27.778140+0000","last_peered":"2024-09-10T20:23:27.778140+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:27.778140+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46709328,"num_objects":5702,"num_object_clones":0,"num_object_copies":22808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":45615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5776","reported_seq":5839,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.597741+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:25.597741+0000","last_peered":"2024-09-10T20:23:25.597741+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:25.597741+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":46188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'5979","reported_seq":6070,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:27.776219+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:27.776219+0000","last_peered":"2024-09-10T20:23:27.776219+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:27.776219+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5979,"log_dups_size":0,"ondisk_log_size":5979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48952480,"num_objects":5976,"num_object_clones":0,"num_object_copies":23904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":47813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5757","reported_seq":5833,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:25.111343+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:25.111343+0000","last_peered":"2024-09-10T20:23:25.111343+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:25.111343+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5755,"num_write_kb":46028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":761522320,"num_objects":92963,"num_object_clones":0,"num_object_copies":371852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80258,"num_write_kb":641984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1420947456,"data_stored":1420947456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93024,"ondisk_log_size":93024,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709124,"kb_used_data":370608,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93662716,"statfs":{"total":96636764160,"available":95910621184,"internally_reserved":0,"allocated":379502592,"data_stored":379298912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715084,"kb_used_data":372924,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93656756,"statfs":{"total":96636764160,"available":95904518144,"internally_reserved":0,"allocated":381874176,"data_stored":381670496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,3,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507500,"kb_used_data":273116,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93864340,"statfs":{"total":96636764160,"available":96117084160,"internally_reserved":0,"allocated":279670784,"data_stored":279467104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708964,"kb_used_data":370516,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93662876,"statfs":{"total":96636764160,"available":95910785024,"internally_reserved":0,"allocated":379408384,"data_stored":379204704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379121664,"data_stored":379121664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279638016,"data_stored":279638016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382464000,"data_stored":382464000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379723776,"data_stored":379723776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:31.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 230 20 94470 94450 3.20775 2.375 0.0604126 0.0486945 2024-09-10T20:23:32.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 231 20 94778 94758 3.20428 2.40625 0.0392292 0.0487492 2024-09-10T20:23:33.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 232 20 95083 95063 3.20074 2.38281 0.10548 0.0488045 2024-09-10T20:23:34.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:34.628 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:34.628 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:34.656 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-10T20:23:33.957551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":773204112,"num_objects":94389,"num_object_clones":0,"num_object_copies":377556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81684,"num_write_kb":653392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94450,"ondisk_log_size":94450,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2697380,"kb_used_data":1409188,"kb_used_omap":14,"kb_used_meta":1288113,"kb_avail":374789980,"statfs":{"total":386547056640,"available":383784939520,"internally_reserved":0,"allocated":1443008512,"data_stored":1442193792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1319028278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,16,4,13,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28363616,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2274,"num_write_kb":-18181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1875,"num_bytes_recovered":-15327232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004598"},"pg_stats":[{"pgid":"1.f","version":"38'5880","reported_seq":5933,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.522681+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:30.522681+0000","last_peered":"2024-09-10T20:23:30.522681+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:30.522681+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168960,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5880,"num_write_kb":47040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6013","reported_seq":6138,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:33.814467+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:33.814467+0000","last_peered":"2024-09-10T20:23:33.814467+0000","last_clean":"2024-09-10T20:23:33.814467+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:33.814467+0000","last_undegraded":"2024-09-10T20:23:33.814467+0000","last_fullsized":"2024-09-10T20:23:33.814467+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6013,"log_dups_size":0,"ondisk_log_size":6013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49209344,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5929","reported_seq":6018,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.835230+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:32.835230+0000","last_peered":"2024-09-10T20:23:32.835230+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:32.835230+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544336,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":47414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6074","reported_seq":10238,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:33.666926+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:33.666926+0000","last_peered":"2024-09-10T20:23:33.666926+0000","last_clean":"2024-09-10T20:23:33.666926+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:33.666926+0000","last_undegraded":"2024-09-10T20:23:33.666926+0000","last_fullsized":"2024-09-10T20:23:33.666926+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6074,"log_dups_size":0,"ondisk_log_size":6074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49709056,"num_objects":6068,"num_object_clones":0,"num_object_copies":24272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":334,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5861","reported_seq":5925,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.590548+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:30.590548+0000","last_peered":"2024-09-10T20:23:30.590548+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:30.590548+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5861,"log_dups_size":0,"ondisk_log_size":5861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47985824,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5859,"num_write_kb":46869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5846","reported_seq":5900,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.626009+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:30.626009+0000","last_peered":"2024-09-10T20:23:30.626009+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:30.626009+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47890432,"num_objects":5846,"num_object_clones":0,"num_object_copies":23384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5846,"num_write_kb":46768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5839","reported_seq":5916,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.175009+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:30.175009+0000","last_peered":"2024-09-10T20:23:30.175009+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:30.175009+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47781024,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":46676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5926","reported_seq":6002,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.129588+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:30.129588+0000","last_peered":"2024-09-10T20:23:30.129588+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:30.129588+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48501920,"num_objects":5921,"num_object_clones":0,"num_object_copies":23684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":47380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5834","reported_seq":9800,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:30.629118+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:30.629118+0000","last_peered":"2024-09-10T20:23:30.629118+0000","last_clean":"2024-09-10T20:23:30.629118+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:30.629118+0000","last_undegraded":"2024-09-10T20:23:30.629118+0000","last_fullsized":"2024-09-10T20:23:30.629118+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47700560,"num_objects":5823,"num_object_clones":0,"num_object_copies":23292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":19096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5921","reported_seq":6432,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.822237+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:32.822237+0000","last_peered":"2024-09-10T20:23:32.822237+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:32.822237+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48485536,"num_objects":5919,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4226}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5875","reported_seq":5942,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.518319+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:30.518319+0000","last_peered":"2024-09-10T20:23:30.518319+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:30.518319+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48082672,"num_objects":5870,"num_object_clones":0,"num_object_copies":23480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":46964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5880","reported_seq":5934,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.593621+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:30.593621+0000","last_peered":"2024-09-10T20:23:30.593621+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:30.593621+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48159312,"num_objects":5879,"num_object_clones":0,"num_object_copies":23516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":47031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5778","reported_seq":5866,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.798506+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:32.798506+0000","last_peered":"2024-09-10T20:23:32.798506+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:32.798506+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323728,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":46215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5871","reported_seq":5934,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.676999+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:30.676999+0000","last_peered":"2024-09-10T20:23:30.676999+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:30.676999+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5871,"log_dups_size":0,"ondisk_log_size":5871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48059552,"num_objects":5867,"num_object_clones":0,"num_object_copies":23468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":46948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3892}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6069","reported_seq":6161,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.894410+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:32.894410+0000","last_peered":"2024-09-10T20:23:32.894410+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:32.894410+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49689760,"num_objects":6066,"num_object_clones":0,"num_object_copies":24264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6067,"num_write_kb":48533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5854","reported_seq":5927,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.165214+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:30.165214+0000","last_peered":"2024-09-10T20:23:30.165214+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:30.165214+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47912096,"num_objects":5849,"num_object_clones":0,"num_object_copies":23396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3870}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":773204112,"num_objects":94389,"num_object_clones":0,"num_object_copies":377556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81684,"num_write_kb":653392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1444769792,"data_stored":1444769792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94450,"ondisk_log_size":94450,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718936,"kb_used_data":376264,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93652904,"statfs":{"total":96636764160,"available":95900573696,"internally_reserved":0,"allocated":385294336,"data_stored":385090656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724896,"kb_used_data":378576,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93646944,"statfs":{"total":96636764160,"available":95894470656,"internally_reserved":0,"allocated":387661824,"data_stored":387458144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,1,8,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535792,"kb_used_data":279200,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93836048,"statfs":{"total":96636764160,"available":96088113152,"internally_reserved":0,"allocated":285900800,"data_stored":285697120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":262729426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717756,"kb_used_data":375148,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93654084,"statfs":{"total":96636764160,"available":95901782016,"internally_reserved":0,"allocated":384151552,"data_stored":383947872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350809807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385007616,"data_stored":385007616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285847552,"data_stored":285847552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388345856,"data_stored":388345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385568768,"data_stored":385568768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:34.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:34.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 233 20 95361 95341 3.19632 2.17188 0.0805168 0.0488735 2024-09-10T20:23:35.297 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:35.297 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:35.325 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-10T20:23:33.957551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":773204112,"num_objects":94389,"num_object_clones":0,"num_object_copies":377556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81684,"num_write_kb":653392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94450,"ondisk_log_size":94450,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2697380,"kb_used_data":1409188,"kb_used_omap":14,"kb_used_meta":1288113,"kb_avail":374789980,"statfs":{"total":386547056640,"available":383784939520,"internally_reserved":0,"allocated":1443008512,"data_stored":1442193792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1319028278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,16,4,13,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28363616,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2274,"num_write_kb":-18181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1875,"num_bytes_recovered":-15327232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004598"},"pg_stats":[{"pgid":"1.f","version":"38'5880","reported_seq":5933,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.522681+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:30.522681+0000","last_peered":"2024-09-10T20:23:30.522681+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:30.522681+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168960,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5880,"num_write_kb":47040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6013","reported_seq":6138,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:33.814467+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:33.814467+0000","last_peered":"2024-09-10T20:23:33.814467+0000","last_clean":"2024-09-10T20:23:33.814467+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:33.814467+0000","last_undegraded":"2024-09-10T20:23:33.814467+0000","last_fullsized":"2024-09-10T20:23:33.814467+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6013,"log_dups_size":0,"ondisk_log_size":6013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49209344,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5929","reported_seq":6018,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.835230+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:32.835230+0000","last_peered":"2024-09-10T20:23:32.835230+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:32.835230+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544336,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":47414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6074","reported_seq":10238,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:33.666926+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:33.666926+0000","last_peered":"2024-09-10T20:23:33.666926+0000","last_clean":"2024-09-10T20:23:33.666926+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:33.666926+0000","last_undegraded":"2024-09-10T20:23:33.666926+0000","last_fullsized":"2024-09-10T20:23:33.666926+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6074,"log_dups_size":0,"ondisk_log_size":6074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49709056,"num_objects":6068,"num_object_clones":0,"num_object_copies":24272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":334,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5861","reported_seq":5925,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.590548+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:30.590548+0000","last_peered":"2024-09-10T20:23:30.590548+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:30.590548+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5861,"log_dups_size":0,"ondisk_log_size":5861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47985824,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5859,"num_write_kb":46869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5846","reported_seq":5900,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.626009+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:30.626009+0000","last_peered":"2024-09-10T20:23:30.626009+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:30.626009+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47890432,"num_objects":5846,"num_object_clones":0,"num_object_copies":23384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5846,"num_write_kb":46768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5839","reported_seq":5916,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.175009+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:30.175009+0000","last_peered":"2024-09-10T20:23:30.175009+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:30.175009+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47781024,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":46676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5926","reported_seq":6002,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.129588+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:30.129588+0000","last_peered":"2024-09-10T20:23:30.129588+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:30.129588+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48501920,"num_objects":5921,"num_object_clones":0,"num_object_copies":23684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":47380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5834","reported_seq":9800,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:30.629118+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:30.629118+0000","last_peered":"2024-09-10T20:23:30.629118+0000","last_clean":"2024-09-10T20:23:30.629118+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:30.629118+0000","last_undegraded":"2024-09-10T20:23:30.629118+0000","last_fullsized":"2024-09-10T20:23:30.629118+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47700560,"num_objects":5823,"num_object_clones":0,"num_object_copies":23292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":19096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5921","reported_seq":6432,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.822237+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:32.822237+0000","last_peered":"2024-09-10T20:23:32.822237+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:32.822237+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48485536,"num_objects":5919,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4226}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5875","reported_seq":5942,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.518319+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:30.518319+0000","last_peered":"2024-09-10T20:23:30.518319+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:30.518319+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48082672,"num_objects":5870,"num_object_clones":0,"num_object_copies":23480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":46964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5880","reported_seq":5934,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.593621+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:30.593621+0000","last_peered":"2024-09-10T20:23:30.593621+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:30.593621+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48159312,"num_objects":5879,"num_object_clones":0,"num_object_copies":23516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":47031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5778","reported_seq":5866,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.798506+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:32.798506+0000","last_peered":"2024-09-10T20:23:32.798506+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:32.798506+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323728,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":46215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5871","reported_seq":5934,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.676999+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:30.676999+0000","last_peered":"2024-09-10T20:23:30.676999+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:30.676999+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5871,"log_dups_size":0,"ondisk_log_size":5871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48059552,"num_objects":5867,"num_object_clones":0,"num_object_copies":23468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":46948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3892}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6069","reported_seq":6161,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.894410+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:32.894410+0000","last_peered":"2024-09-10T20:23:32.894410+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:32.894410+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49689760,"num_objects":6066,"num_object_clones":0,"num_object_copies":24264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6067,"num_write_kb":48533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5854","reported_seq":5927,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:30.165214+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:30.165214+0000","last_peered":"2024-09-10T20:23:30.165214+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:30.165214+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47912096,"num_objects":5849,"num_object_clones":0,"num_object_copies":23396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3870}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":773204112,"num_objects":94389,"num_object_clones":0,"num_object_copies":377556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81684,"num_write_kb":653392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1444769792,"data_stored":1444769792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94450,"ondisk_log_size":94450,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718936,"kb_used_data":376264,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93652904,"statfs":{"total":96636764160,"available":95900573696,"internally_reserved":0,"allocated":385294336,"data_stored":385090656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724896,"kb_used_data":378576,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93646944,"statfs":{"total":96636764160,"available":95894470656,"internally_reserved":0,"allocated":387661824,"data_stored":387458144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,1,8,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535792,"kb_used_data":279200,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93836048,"statfs":{"total":96636764160,"available":96088113152,"internally_reserved":0,"allocated":285900800,"data_stored":285697120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":262729426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717756,"kb_used_data":375148,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93654084,"statfs":{"total":96636764160,"available":95901782016,"internally_reserved":0,"allocated":384151552,"data_stored":383947872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350809807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385007616,"data_stored":385007616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285847552,"data_stored":285847552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388345856,"data_stored":388345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385568768,"data_stored":385568768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'5880", 'reported_seq': 5933, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.522681+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:30.522681+0000', 'last_peered': '2024-09-10T20:23:30.522681+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:30.522681+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5880, 'log_dups_size': 0, 'ondisk_log_size': 5880, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48168960, 'num_objects': 5880, 'num_object_clones': 0, 'num_object_copies': 23520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5880, 'num_write_kb': 47040, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3914}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'5929", 'reported_seq': 6018, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:32.835230+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:32.835230+0000', 'last_peered': '2024-09-10T20:23:32.835230+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:32.835230+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5929, 'log_dups_size': 0, 'ondisk_log_size': 5929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48544336, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5928, 'num_write_kb': 47414, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3964}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'5861", 'reported_seq': 5925, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.590548+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:30.590548+0000', 'last_peered': '2024-09-10T20:23:30.590548+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:30.590548+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5861, 'log_dups_size': 0, 'ondisk_log_size': 5861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47985824, 'num_objects': 5858, 'num_object_clones': 0, 'num_object_copies': 23432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5859, 'num_write_kb': 46869, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3878}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:35.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'5846", 'reported_seq': 5900, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.626009+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:30.626009+0000', 'last_peered': '2024-09-10T20:23:30.626009+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:30.626009+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5846, 'log_dups_size': 0, 'ondisk_log_size': 5846, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47890432, 'num_objects': 5846, 'num_object_clones': 0, 'num_object_copies': 23384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5846, 'num_write_kb': 46768, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3908}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'5839", 'reported_seq': 5916, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.175009+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:30.175009+0000', 'last_peered': '2024-09-10T20:23:30.175009+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:30.175009+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5839, 'log_dups_size': 0, 'ondisk_log_size': 5839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47781024, 'num_objects': 5833, 'num_object_clones': 0, 'num_object_copies': 23332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5837, 'num_write_kb': 46676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3880}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'5926", 'reported_seq': 6002, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.129588+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:30.129588+0000', 'last_peered': '2024-09-10T20:23:30.129588+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:30.129588+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5926, 'log_dups_size': 0, 'ondisk_log_size': 5926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48501920, 'num_objects': 5921, 'num_object_clones': 0, 'num_object_copies': 23684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5924, 'num_write_kb': 47380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3878}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'5921", 'reported_seq': 6432, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:32.822237+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:32.822237+0000', 'last_peered': '2024-09-10T20:23:32.822237+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:32.822237+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5921, 'log_dups_size': 0, 'ondisk_log_size': 5921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48485536, 'num_objects': 5919, 'num_object_clones': 0, 'num_object_copies': 23676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1693, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5919, 'num_write_kb': 47350, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 212, 'num_bytes_recovered': 1720320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4226}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'5875", 'reported_seq': 5942, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.518319+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:30.518319+0000', 'last_peered': '2024-09-10T20:23:30.518319+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:30.518319+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5875, 'log_dups_size': 0, 'ondisk_log_size': 5875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48082672, 'num_objects': 5870, 'num_object_clones': 0, 'num_object_copies': 23480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5872, 'num_write_kb': 46964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3914}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'5880", 'reported_seq': 5934, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.593621+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:30.593621+0000', 'last_peered': '2024-09-10T20:23:30.593621+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:30.593621+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5880, 'log_dups_size': 0, 'ondisk_log_size': 5880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48159312, 'num_objects': 5879, 'num_object_clones': 0, 'num_object_copies': 23516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5879, 'num_write_kb': 47031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:35.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:35.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'5778", 'reported_seq': 5866, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:32.798506+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:32.798506+0000', 'last_peered': '2024-09-10T20:23:32.798506+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:32.798506+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5778, 'log_dups_size': 0, 'ondisk_log_size': 5778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47323728, 'num_objects': 5777, 'num_object_clones': 0, 'num_object_copies': 23108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5777, 'num_write_kb': 46215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:35.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:35.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'5871", 'reported_seq': 5934, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.676999+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:30.676999+0000', 'last_peered': '2024-09-10T20:23:30.676999+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:30.676999+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5871, 'log_dups_size': 0, 'ondisk_log_size': 5871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48059552, 'num_objects': 5867, 'num_object_clones': 0, 'num_object_copies': 23468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5869, 'num_write_kb': 46948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:35.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:35.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6069", 'reported_seq': 6161, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:32.894410+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:32.894410+0000', 'last_peered': '2024-09-10T20:23:32.894410+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:32.894410+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6069, 'log_dups_size': 0, 'ondisk_log_size': 6069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49689760, 'num_objects': 6066, 'num_object_clones': 0, 'num_object_copies': 24264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6067, 'num_write_kb': 48533, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:35.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:35.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'5854", 'reported_seq': 5927, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:30.165214+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:30.165214+0000', 'last_peered': '2024-09-10T20:23:30.165214+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:30.165214+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5854, 'log_dups_size': 0, 'ondisk_log_size': 5854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47912096, 'num_objects': 5849, 'num_object_clones': 0, 'num_object_copies': 23396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5852, 'num_write_kb': 46804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:35.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:35.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 234 20 95629 95609 3.19161 2.09375 0.0602811 0.0489436 2024-09-10T20:23:36.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:36.109 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25351/377556 objects degraded (6.715%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":304,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94389,"data_bytes":773204112,"bytes_used":2762117120,"bytes_avail":383784939520,"bytes_total":386547056640,"degraded_objects":25351,"degraded_total":377556,"degraded_ratio":0.067145006303700641},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:36.132 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:23:36.133 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:36.742 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:36.742 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:36.772 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-10T20:23:35.958045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":780961936,"num_objects":95336,"num_object_clones":0,"num_object_copies":381344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82631,"num_write_kb":660968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95397,"ondisk_log_size":95397,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718520,"kb_used_data":1422008,"kb_used_omap":14,"kb_used_meta":1296433,"kb_avail":374768840,"statfs":{"total":386547056640,"available":383763292160,"internally_reserved":0,"allocated":1456136192,"data_stored":1455321472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1327547958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,12,5,14,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31531008,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3849,"num_write_kb":30792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":696320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004370"},"pg_stats":[{"pgid":"1.f","version":"38'5972","reported_seq":6026,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.589587+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:35.589587+0000","last_peered":"2024-09-10T20:23:35.589587+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:35.589587+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5972,"log_dups_size":0,"ondisk_log_size":5972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48922624,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6013","reported_seq":6138,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:33.814467+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:33.814467+0000","last_peered":"2024-09-10T20:23:33.814467+0000","last_clean":"2024-09-10T20:23:33.814467+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:33.814467+0000","last_undegraded":"2024-09-10T20:23:33.814467+0000","last_fullsized":"2024-09-10T20:23:33.814467+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6013,"log_dups_size":0,"ondisk_log_size":6013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49209344,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5929","reported_seq":6018,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.835230+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:32.835230+0000","last_peered":"2024-09-10T20:23:32.835230+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:32.835230+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544336,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":47414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6074","reported_seq":10238,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:33.666926+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:33.666926+0000","last_peered":"2024-09-10T20:23:33.666926+0000","last_clean":"2024-09-10T20:23:33.666926+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:33.666926+0000","last_undegraded":"2024-09-10T20:23:33.666926+0000","last_fullsized":"2024-09-10T20:23:33.666926+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6074,"log_dups_size":0,"ondisk_log_size":6074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49709056,"num_objects":6068,"num_object_clones":0,"num_object_copies":24272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":334,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5967","reported_seq":6031,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.712190+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:35.712190+0000","last_peered":"2024-09-10T20:23:35.712190+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:35.712190+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48854176,"num_objects":5964,"num_object_clones":0,"num_object_copies":23856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":47717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5936","reported_seq":5990,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.612111+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:35.612111+0000","last_peered":"2024-09-10T20:23:35.612111+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:35.612111+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48627712,"num_objects":5936,"num_object_clones":0,"num_object_copies":23744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5936","reported_seq":6014,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.196657+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:35.196657+0000","last_peered":"2024-09-10T20:23:35.196657+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:35.196657+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48575648,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5934,"num_write_kb":47452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3977}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6029","reported_seq":6105,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.232014+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:35.232014+0000","last_peered":"2024-09-10T20:23:35.232014+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:35.232014+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49345696,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":48204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5914","reported_seq":9880,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:35.612384+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:35.612384+0000","last_peered":"2024-09-10T20:23:35.612384+0000","last_clean":"2024-09-10T20:23:35.612384+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:35.612384+0000","last_undegraded":"2024-09-10T20:23:35.612384+0000","last_fullsized":"2024-09-10T20:23:35.612384+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48355920,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2467,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5921","reported_seq":6432,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.822237+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:32.822237+0000","last_peered":"2024-09-10T20:23:32.822237+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:32.822237+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48485536,"num_objects":5919,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4226}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5966","reported_seq":6033,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.574857+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:35.574857+0000","last_peered":"2024-09-10T20:23:35.574857+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:35.574857+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48828144,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":47692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5975","reported_seq":6031,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.578687+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:35.578687+0000","last_peered":"2024-09-10T20:23:35.578687+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:35.578687+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48937552,"num_objects":5974,"num_object_clones":0,"num_object_copies":23896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":47791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5778","reported_seq":5866,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.798506+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:32.798506+0000","last_peered":"2024-09-10T20:23:32.798506+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:32.798506+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323728,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":46215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5965","reported_seq":6028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.653660+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:35.653660+0000","last_peered":"2024-09-10T20:23:35.653660+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:35.653660+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48829600,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":47700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3986}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6069","reported_seq":6161,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:32.894410+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:32.894410+0000","last_peered":"2024-09-10T20:23:32.894410+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:32.894410+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49689760,"num_objects":6066,"num_object_clones":0,"num_object_copies":24264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6067,"num_write_kb":48533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5953","reported_seq":6028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.241212+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:35.241212+0000","last_peered":"2024-09-10T20:23:35.241212+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:35.241212+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48723104,"num_objects":5948,"num_object_clones":0,"num_object_copies":23792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":780961936,"num_objects":95336,"num_object_clones":0,"num_object_copies":381344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82631,"num_write_kb":660968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1456852992,"data_stored":1456852992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95397,"ondisk_log_size":95397,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728928,"kb_used_data":382096,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93642912,"statfs":{"total":96636764160,"available":95890341888,"internally_reserved":0,"allocated":391266304,"data_stored":391062624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724896,"kb_used_data":378576,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93646944,"statfs":{"total":96636764160,"available":95894470656,"internally_reserved":0,"allocated":387661824,"data_stored":387458144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,1,8,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535792,"kb_used_data":279200,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93836048,"statfs":{"total":96636764160,"available":96088113152,"internally_reserved":0,"allocated":285900800,"data_stored":285697120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":262729426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728904,"kb_used_data":382136,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93642936,"statfs":{"total":96636764160,"available":95890366464,"internally_reserved":0,"allocated":391307264,"data_stored":391103584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,2,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391061504,"data_stored":391061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285847552,"data_stored":285847552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388345856,"data_stored":388345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391598080,"data_stored":391598080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:36.803 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 235 20 95894 95874 3.18684 2.07031 0.0554362 0.0490182 2024-09-10T20:23:37.804 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 236 20 96199 96179 3.18342 2.38281 0.0476823 0.0490709 2024-09-10T20:23:38.804 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 237 20 96423 96403 3.17737 1.75 0.0571385 0.0491645 2024-09-10T20:23:39.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:39.804 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 238 20 96674 96654 3.17226 1.96094 0.142161 0.0492425 2024-09-10T20:23:40.379 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:40.405 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-10T20:23:39.959212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":785139856,"num_objects":95846,"num_object_clones":0,"num_object_copies":383384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83141,"num_write_kb":665048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95907,"ondisk_log_size":95907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704792,"kb_used_data":1433240,"kb_used_omap":14,"kb_used_meta":1271473,"kb_avail":374782568,"statfs":{"total":386547056640,"available":383777349632,"internally_reserved":0,"allocated":1467637760,"data_stored":1466823040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1301988918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,6,7,7,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25034752,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003945"},"pg_stats":[{"pgid":"1.f","version":"38'5972","reported_seq":6026,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.589587+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:35.589587+0000","last_peered":"2024-09-10T20:23:35.589587+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:35.589587+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5972,"log_dups_size":0,"ondisk_log_size":5972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48922624,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6097","reported_seq":6223,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:38.717833+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:38.717833+0000","last_peered":"2024-09-10T20:23:38.717833+0000","last_clean":"2024-09-10T20:23:38.717833+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:38.717833+0000","last_undegraded":"2024-09-10T20:23:38.717833+0000","last_fullsized":"2024-09-10T20:23:38.717833+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49897472,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6016","reported_seq":6103,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.901343+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:37.901343+0000","last_peered":"2024-09-10T20:23:37.901343+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:37.901343+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49257040,"num_objects":6013,"num_object_clones":0,"num_object_copies":24052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":48110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6161","reported_seq":10324,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:38.826830+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:38.826830+0000","last_peered":"2024-09-10T20:23:38.826830+0000","last_clean":"2024-09-10T20:23:38.826830+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:38.826830+0000","last_undegraded":"2024-09-10T20:23:38.826830+0000","last_fullsized":"2024-09-10T20:23:38.826830+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50421760,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":421,"num_write_kb":3368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5967","reported_seq":6031,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.712190+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:35.712190+0000","last_peered":"2024-09-10T20:23:35.712190+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:35.712190+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48854176,"num_objects":5964,"num_object_clones":0,"num_object_copies":23856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":47717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5936","reported_seq":5990,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.612111+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:35.612111+0000","last_peered":"2024-09-10T20:23:35.612111+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:35.612111+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48627712,"num_objects":5936,"num_object_clones":0,"num_object_copies":23744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5936","reported_seq":6014,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.196657+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:35.196657+0000","last_peered":"2024-09-10T20:23:35.196657+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:35.196657+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48575648,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5934,"num_write_kb":47452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3977}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6029","reported_seq":6105,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.232014+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:35.232014+0000","last_peered":"2024-09-10T20:23:35.232014+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:35.232014+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49345696,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":48204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5914","reported_seq":9880,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:35.612384+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:35.612384+0000","last_peered":"2024-09-10T20:23:35.612384+0000","last_clean":"2024-09-10T20:23:35.612384+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:35.612384+0000","last_undegraded":"2024-09-10T20:23:35.612384+0000","last_fullsized":"2024-09-10T20:23:35.612384+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48355920,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2467,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6003","reported_seq":6612,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.923584+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:37.923584+0000","last_peered":"2024-09-10T20:23:37.923584+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:37.923584+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6003,"log_dups_size":0,"ondisk_log_size":6003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157280,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6001,"num_write_kb":48006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5966","reported_seq":6033,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.574857+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:35.574857+0000","last_peered":"2024-09-10T20:23:35.574857+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:35.574857+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48828144,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":47692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5975","reported_seq":6031,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.578687+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:35.578687+0000","last_peered":"2024-09-10T20:23:35.578687+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:35.578687+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48937552,"num_objects":5974,"num_object_clones":0,"num_object_copies":23896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":47791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5858","reported_seq":5946,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.736304+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:37.736304+0000","last_peered":"2024-09-10T20:23:37.736304+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:37.736304+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47979088,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":46855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5965","reported_seq":6028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.653660+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:35.653660+0000","last_peered":"2024-09-10T20:23:35.653660+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:35.653660+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48829600,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":47700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3986}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6159","reported_seq":6251,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.893680+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:37.893680+0000","last_peered":"2024-09-10T20:23:37.893680+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:37.893680+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427040,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5953","reported_seq":6028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.241212+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:35.241212+0000","last_peered":"2024-09-10T20:23:35.241212+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:35.241212+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48723104,"num_objects":5948,"num_object_clones":0,"num_object_copies":23792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":785139856,"num_objects":95846,"num_object_clones":0,"num_object_copies":383384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83141,"num_write_kb":665048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1468575744,"data_stored":1468575744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95907,"ondisk_log_size":95907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728928,"kb_used_data":382096,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93642912,"statfs":{"total":96636764160,"available":95890341888,"internally_reserved":0,"allocated":391266304,"data_stored":391062624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718064,"kb_used_data":384224,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93653776,"statfs":{"total":96636764160,"available":95901466624,"internally_reserved":0,"allocated":393445376,"data_stored":393241696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528896,"kb_used_data":284784,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93842944,"statfs":{"total":96636764160,"available":96095174656,"internally_reserved":0,"allocated":291618816,"data_stored":291415136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":249949906},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728904,"kb_used_data":382136,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93642936,"statfs":{"total":96636764160,"available":95890366464,"internally_reserved":0,"allocated":391307264,"data_stored":391103584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,2,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391061504,"data_stored":391061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291717120,"data_stored":291717120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":394199040,"data_stored":394199040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391598080,"data_stored":391598080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:40.406 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:40.804 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 239 20 96925 96905 3.16719 1.96094 0.108843 0.0493194 2024-09-10T20:23:41.043 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:41.043 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:41.075 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-10T20:23:39.959212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":785139856,"num_objects":95846,"num_object_clones":0,"num_object_copies":383384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83141,"num_write_kb":665048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95907,"ondisk_log_size":95907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704792,"kb_used_data":1433240,"kb_used_omap":14,"kb_used_meta":1271473,"kb_avail":374782568,"statfs":{"total":386547056640,"available":383777349632,"internally_reserved":0,"allocated":1467637760,"data_stored":1466823040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1301988918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,6,7,7,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25034752,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003945"},"pg_stats":[{"pgid":"1.f","version":"38'5972","reported_seq":6026,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.589587+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:35.589587+0000","last_peered":"2024-09-10T20:23:35.589587+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:35.589587+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5972,"log_dups_size":0,"ondisk_log_size":5972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48922624,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6097","reported_seq":6223,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:38.717833+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:38.717833+0000","last_peered":"2024-09-10T20:23:38.717833+0000","last_clean":"2024-09-10T20:23:38.717833+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:38.717833+0000","last_undegraded":"2024-09-10T20:23:38.717833+0000","last_fullsized":"2024-09-10T20:23:38.717833+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49897472,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6016","reported_seq":6103,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.901343+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:37.901343+0000","last_peered":"2024-09-10T20:23:37.901343+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:37.901343+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49257040,"num_objects":6013,"num_object_clones":0,"num_object_copies":24052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":48110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6161","reported_seq":10324,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:38.826830+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:38.826830+0000","last_peered":"2024-09-10T20:23:38.826830+0000","last_clean":"2024-09-10T20:23:38.826830+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:38.826830+0000","last_undegraded":"2024-09-10T20:23:38.826830+0000","last_fullsized":"2024-09-10T20:23:38.826830+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50421760,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":421,"num_write_kb":3368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'5967","reported_seq":6031,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.712190+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:35.712190+0000","last_peered":"2024-09-10T20:23:35.712190+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:35.712190+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48854176,"num_objects":5964,"num_object_clones":0,"num_object_copies":23856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":47717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'5936","reported_seq":5990,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.612111+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:35.612111+0000","last_peered":"2024-09-10T20:23:35.612111+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:35.612111+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48627712,"num_objects":5936,"num_object_clones":0,"num_object_copies":23744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'5936","reported_seq":6014,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.196657+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:35.196657+0000","last_peered":"2024-09-10T20:23:35.196657+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:35.196657+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48575648,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5934,"num_write_kb":47452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3977}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6029","reported_seq":6105,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.232014+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:35.232014+0000","last_peered":"2024-09-10T20:23:35.232014+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:35.232014+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49345696,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":48204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5914","reported_seq":9880,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:35.612384+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:35.612384+0000","last_peered":"2024-09-10T20:23:35.612384+0000","last_clean":"2024-09-10T20:23:35.612384+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:35.612384+0000","last_undegraded":"2024-09-10T20:23:35.612384+0000","last_fullsized":"2024-09-10T20:23:35.612384+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48355920,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2467,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6003","reported_seq":6612,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.923584+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:37.923584+0000","last_peered":"2024-09-10T20:23:37.923584+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:37.923584+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6003,"log_dups_size":0,"ondisk_log_size":6003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157280,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6001,"num_write_kb":48006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'5966","reported_seq":6033,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.574857+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:35.574857+0000","last_peered":"2024-09-10T20:23:35.574857+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:35.574857+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48828144,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":47692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'5975","reported_seq":6031,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.578687+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:35.578687+0000","last_peered":"2024-09-10T20:23:35.578687+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:35.578687+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48937552,"num_objects":5974,"num_object_clones":0,"num_object_copies":23896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":47791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5858","reported_seq":5946,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.736304+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:37.736304+0000","last_peered":"2024-09-10T20:23:37.736304+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:37.736304+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47979088,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":46855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'5965","reported_seq":6028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.653660+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:35.653660+0000","last_peered":"2024-09-10T20:23:35.653660+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:35.653660+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48829600,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":47700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3986}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6159","reported_seq":6251,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.893680+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:37.893680+0000","last_peered":"2024-09-10T20:23:37.893680+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:37.893680+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427040,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'5953","reported_seq":6028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:35.241212+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:35.241212+0000","last_peered":"2024-09-10T20:23:35.241212+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:35.241212+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48723104,"num_objects":5948,"num_object_clones":0,"num_object_copies":23792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":785139856,"num_objects":95846,"num_object_clones":0,"num_object_copies":383384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83141,"num_write_kb":665048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1468575744,"data_stored":1468575744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95907,"ondisk_log_size":95907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728928,"kb_used_data":382096,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93642912,"statfs":{"total":96636764160,"available":95890341888,"internally_reserved":0,"allocated":391266304,"data_stored":391062624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718064,"kb_used_data":384224,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93653776,"statfs":{"total":96636764160,"available":95901466624,"internally_reserved":0,"allocated":393445376,"data_stored":393241696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528896,"kb_used_data":284784,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93842944,"statfs":{"total":96636764160,"available":96095174656,"internally_reserved":0,"allocated":291618816,"data_stored":291415136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":249949906},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728904,"kb_used_data":382136,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93642936,"statfs":{"total":96636764160,"available":95890366464,"internally_reserved":0,"allocated":391307264,"data_stored":391103584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,2,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391061504,"data_stored":391061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291717120,"data_stored":291717120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":394199040,"data_stored":394199040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391598080,"data_stored":391598080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:41.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:41.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'5972", 'reported_seq': 6026, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.589587+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:35.589587+0000', 'last_peered': '2024-09-10T20:23:35.589587+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:35.589587+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5972, 'log_dups_size': 0, 'ondisk_log_size': 5972, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48922624, 'num_objects': 5972, 'num_object_clones': 0, 'num_object_copies': 23888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5972, 'num_write_kb': 47776, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4006}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:41.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:41.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6016", 'reported_seq': 6103, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:37.901343+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:37.901343+0000', 'last_peered': '2024-09-10T20:23:37.901343+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:37.901343+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6016, 'log_dups_size': 0, 'ondisk_log_size': 6016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49257040, 'num_objects': 6013, 'num_object_clones': 0, 'num_object_copies': 24052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6015, 'num_write_kb': 48110, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4051}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:41.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:41.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'5967", 'reported_seq': 6031, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.712190+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:35.712190+0000', 'last_peered': '2024-09-10T20:23:35.712190+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:35.712190+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5967, 'log_dups_size': 0, 'ondisk_log_size': 5967, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48854176, 'num_objects': 5964, 'num_object_clones': 0, 'num_object_copies': 23856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5965, 'num_write_kb': 47717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3984}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:41.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:41.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'5936", 'reported_seq': 5990, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.612111+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:35.612111+0000', 'last_peered': '2024-09-10T20:23:35.612111+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:35.612111+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5936, 'log_dups_size': 0, 'ondisk_log_size': 5936, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48627712, 'num_objects': 5936, 'num_object_clones': 0, 'num_object_copies': 23744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5936, 'num_write_kb': 47488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:41.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:41.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'5936", 'reported_seq': 6014, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.196657+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:35.196657+0000', 'last_peered': '2024-09-10T20:23:35.196657+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:35.196657+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5936, 'log_dups_size': 0, 'ondisk_log_size': 5936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48575648, 'num_objects': 5930, 'num_object_clones': 0, 'num_object_copies': 23720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5934, 'num_write_kb': 47452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3977}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:41.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:41.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6029", 'reported_seq': 6105, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.232014+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:35.232014+0000', 'last_peered': '2024-09-10T20:23:35.232014+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:35.232014+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6029, 'log_dups_size': 0, 'ondisk_log_size': 6029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49345696, 'num_objects': 6024, 'num_object_clones': 0, 'num_object_copies': 24096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6027, 'num_write_kb': 48204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3981}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:41.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:41.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6003", 'reported_seq': 6612, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:37.923584+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:37.923584+0000', 'last_peered': '2024-09-10T20:23:37.923584+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:37.923584+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6003, 'log_dups_size': 0, 'ondisk_log_size': 6003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49157280, 'num_objects': 6001, 'num_object_clones': 0, 'num_object_copies': 24004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1644, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6001, 'num_write_kb': 48006, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 261, 'num_bytes_recovered': 2121728, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4357}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:41.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:41.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'5966", 'reported_seq': 6033, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.574857+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:35.574857+0000', 'last_peered': '2024-09-10T20:23:35.574857+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:35.574857+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5966, 'log_dups_size': 0, 'ondisk_log_size': 5966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48828144, 'num_objects': 5961, 'num_object_clones': 0, 'num_object_copies': 23844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5963, 'num_write_kb': 47692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4005}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:41.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:41.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'5975", 'reported_seq': 6031, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.578687+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:35.578687+0000', 'last_peered': '2024-09-10T20:23:35.578687+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:35.578687+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5975, 'log_dups_size': 0, 'ondisk_log_size': 5975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48937552, 'num_objects': 5974, 'num_object_clones': 0, 'num_object_copies': 23896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5974, 'num_write_kb': 47791, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3996}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:41.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:41.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'5858", 'reported_seq': 5946, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:37.736304+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:37.736304+0000', 'last_peered': '2024-09-10T20:23:37.736304+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:37.736304+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5858, 'log_dups_size': 0, 'ondisk_log_size': 5858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47979088, 'num_objects': 5857, 'num_object_clones': 0, 'num_object_copies': 23428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5857, 'num_write_kb': 46855, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:41.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:41.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'5965", 'reported_seq': 6028, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.653660+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:35.653660+0000', 'last_peered': '2024-09-10T20:23:35.653660+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:35.653660+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5965, 'log_dups_size': 0, 'ondisk_log_size': 5965, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48829600, 'num_objects': 5961, 'num_object_clones': 0, 'num_object_copies': 23844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5963, 'num_write_kb': 47700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3986}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:41.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:41.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6159", 'reported_seq': 6251, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:37.893680+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:37.893680+0000', 'last_peered': '2024-09-10T20:23:37.893680+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:37.893680+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6159, 'log_dups_size': 0, 'ondisk_log_size': 6159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50427040, 'num_objects': 6156, 'num_object_clones': 0, 'num_object_copies': 24624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6157, 'num_write_kb': 49253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:41.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:41.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'5953", 'reported_seq': 6028, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:35.241212+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:35.241212+0000', 'last_peered': '2024-09-10T20:23:35.241212+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:35.241212+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5953, 'log_dups_size': 0, 'ondisk_log_size': 5953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48723104, 'num_objects': 5948, 'num_object_clones': 0, 'num_object_copies': 23792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5951, 'num_write_kb': 47596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3969}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:41.082 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:41.779 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:41.779 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25302/383384 objects degraded (6.600%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":310,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95846,"data_bytes":785139856,"bytes_used":2769707008,"bytes_avail":383777349632,"bytes_total":386547056640,"degraded_objects":25302,"degraded_total":383384,"degraded_ratio":0.065996494376395473,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":60737,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"write_bytes_sec":2085543,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:41.799 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:41.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:41.804 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:2024-09-10T20:23:41.802169+0000 min lat: 0.00230373 max lat: 12.2408 avg lat: 0.0493898 2024-09-10T20:23:41.804 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:23:41.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout: 240 12 97184 97172 3.16269 2.08594 0.0632545 0.0493898 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Total time run: 240.059 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Total writes made: 97184 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Write size: 8192 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Object size: 8192 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Bandwidth (MB/sec): 3.16276 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Stddev Bandwidth: 2.0568 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Max bandwidth (MB/sec): 11.3438 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Min bandwidth (MB/sec): 0 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Average IOPS: 404 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Stddev IOPS: 263.275 2024-09-10T20:23:42.805 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Max IOPS: 1452 2024-09-10T20:23:42.806 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Min IOPS: 0 2024-09-10T20:23:42.806 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Average Latency(s): 0.049394 2024-09-10T20:23:42.806 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Stddev Latency(s): 0.182385 2024-09-10T20:23:42.806 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Max latency(s): 12.2408 2024-09-10T20:23:42.806 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Min latency(s): 0.00230373 2024-09-10T20:23:42.817 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T20:23:42.928 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:42.929 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:42.956 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-10T20:23:41.959853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791939216,"num_objects":96676,"num_object_clones":0,"num_object_copies":386704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83971,"num_write_kb":671688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96737,"ondisk_log_size":96737,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2689452,"kb_used_data":1442860,"kb_used_omap":14,"kb_used_meta":1246513,"kb_avail":374797908,"statfs":{"total":386547056640,"available":383793057792,"internally_reserved":0,"allocated":1477488640,"data_stored":1476673920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1276429878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,10,8,10,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30416896,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3713,"num_write_kb":29704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003844"},"pg_stats":[{"pgid":"1.f","version":"38'6041","reported_seq":6095,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.478184+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:40.478184+0000","last_peered":"2024-09-10T20:23:40.478184+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:40.478184+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49487872,"num_objects":6041,"num_object_clones":0,"num_object_copies":24164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":48328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6097","reported_seq":6223,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:38.717833+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:38.717833+0000","last_peered":"2024-09-10T20:23:38.717833+0000","last_clean":"2024-09-10T20:23:38.717833+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:38.717833+0000","last_undegraded":"2024-09-10T20:23:38.717833+0000","last_fullsized":"2024-09-10T20:23:38.717833+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49897472,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6016","reported_seq":6103,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.901343+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:37.901343+0000","last_peered":"2024-09-10T20:23:37.901343+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:37.901343+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49257040,"num_objects":6013,"num_object_clones":0,"num_object_copies":24052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":48110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6161","reported_seq":10324,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:38.826830+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:38.826830+0000","last_peered":"2024-09-10T20:23:38.826830+0000","last_clean":"2024-09-10T20:23:38.826830+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:38.826830+0000","last_undegraded":"2024-09-10T20:23:38.826830+0000","last_fullsized":"2024-09-10T20:23:38.826830+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50421760,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":421,"num_write_kb":3368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6057","reported_seq":6122,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.548919+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:40.548919+0000","last_peered":"2024-09-10T20:23:40.548919+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:40.548919+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6057,"log_dups_size":0,"ondisk_log_size":6057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49591456,"num_objects":6054,"num_object_clones":0,"num_object_copies":24216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6055,"num_write_kb":48437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6025","reported_seq":6079,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.630988+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:40.630988+0000","last_peered":"2024-09-10T20:23:40.630988+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:40.630988+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49356800,"num_objects":6025,"num_object_clones":0,"num_object_copies":24100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6025,"num_write_kb":48200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6015","reported_seq":6093,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.080186+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:40.080186+0000","last_peered":"2024-09-10T20:23:40.080186+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:40.080186+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222816,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6013,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4056}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6133","reported_seq":6210,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.217079+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:40.217079+0000","last_peered":"2024-09-10T20:23:40.217079+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:40.217079+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6133,"log_dups_size":0,"ondisk_log_size":6133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50197664,"num_objects":6128,"num_object_clones":0,"num_object_copies":24512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6131,"num_write_kb":49036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5987","reported_seq":9953,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:40.716275+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:40.716275+0000","last_peered":"2024-09-10T20:23:40.716275+0000","last_clean":"2024-09-10T20:23:40.716275+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:40.716275+0000","last_undegraded":"2024-09-10T20:23:40.716275+0000","last_fullsized":"2024-09-10T20:23:40.716275+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":5987,"log_dups_size":0,"ondisk_log_size":5987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48953936,"num_objects":5976,"num_object_clones":0,"num_object_copies":23904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2540,"num_write_kb":20320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6003","reported_seq":6612,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.923584+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:37.923584+0000","last_peered":"2024-09-10T20:23:37.923584+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:37.923584+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6003,"log_dups_size":0,"ondisk_log_size":6003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157280,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6001,"num_write_kb":48006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6043","reported_seq":6110,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.649861+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:40.649861+0000","last_peered":"2024-09-10T20:23:40.649861+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:40.649861+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49458928,"num_objects":6038,"num_object_clones":0,"num_object_copies":24152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":48308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6063","reported_seq":6119,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.667386+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:40.667386+0000","last_peered":"2024-09-10T20:23:40.667386+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:40.667386+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6063,"log_dups_size":0,"ondisk_log_size":6063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49658448,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6062,"num_write_kb":48495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4084}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5858","reported_seq":5946,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.736304+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:37.736304+0000","last_peered":"2024-09-10T20:23:37.736304+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:37.736304+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47979088,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":46855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6034","reported_seq":6097,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.416766+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:40.416766+0000","last_peered":"2024-09-10T20:23:40.416766+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:40.416766+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6034,"log_dups_size":0,"ondisk_log_size":6034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49394848,"num_objects":6030,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6032,"num_write_kb":48252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6159","reported_seq":6251,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:37.893680+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:37.893680+0000","last_peered":"2024-09-10T20:23:37.893680+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:37.893680+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427040,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6045","reported_seq":6121,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:40.146893+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:40.146893+0000","last_peered":"2024-09-10T20:23:40.146893+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:40.146893+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49476768,"num_objects":6040,"num_object_clones":0,"num_object_copies":24160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6043,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791939216,"num_objects":96676,"num_object_clones":0,"num_object_copies":386704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83971,"num_write_kb":671688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":2121728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479327744,"data_stored":1479327744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96737,"ondisk_log_size":96737,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721712,"kb_used_data":387360,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93650128,"statfs":{"total":96636764160,"available":95897731072,"internally_reserved":0,"allocated":396656640,"data_stored":396452960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718064,"kb_used_data":384224,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93653776,"statfs":{"total":96636764160,"available":95901466624,"internally_reserved":0,"allocated":393445376,"data_stored":393241696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528896,"kb_used_data":284784,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93842944,"statfs":{"total":96636764160,"available":96095174656,"internally_reserved":0,"allocated":291618816,"data_stored":291415136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":249949906},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215160,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720780,"kb_used_data":386492,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93651060,"statfs":{"total":96636764160,"available":95898685440,"internally_reserved":0,"allocated":395767808,"data_stored":395564128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":396455936,"data_stored":396455936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291717120,"data_stored":291717120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":394199040,"data_stored":394199040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":396955648,"data_stored":396955648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:45.958 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:46.418 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:46.418 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:46.445 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-10T20:23:45.961092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791382184,"num_objects":96609,"num_object_clones":0,"num_object_copies":386436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85098,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97864,"ondisk_log_size":97864,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767672,"kb_used_data":1450296,"kb_used_omap":14,"kb_used_meta":1317297,"kb_avail":374719688,"statfs":{"total":386547056640,"available":383712960512,"internally_reserved":0,"allocated":1485103104,"data_stored":1484288384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1348912694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,8,4,8,7,0,3,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18178072,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3414,"num_write_kb":22529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003541"},"pg_stats":[{"pgid":"1.f","version":"38'6126","reported_seq":6186,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.693407+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:45.693407+0000","last_peered":"2024-09-10T20:23:45.693407+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:45.693407+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49119232,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6156","reported_seq":6282,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:43.815124+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:43.815124+0000","last_peered":"2024-09-10T20:23:43.815124+0000","last_clean":"2024-09-10T20:23:43.815124+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:43.815124+0000","last_undegraded":"2024-09-10T20:23:43.815124+0000","last_fullsized":"2024-09-10T20:23:43.815124+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6076","reported_seq":6164,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.969766+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:42.969766+0000","last_peered":"2024-09-10T20:23:42.969766+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:42.969766+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49674856,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6075,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6223","reported_seq":10387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:43.847367+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:43.847367+0000","last_peered":"2024-09-10T20:23:43.847367+0000","last_clean":"2024-09-10T20:23:43.847367+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:43.847367+0000","last_undegraded":"2024-09-10T20:23:43.847367+0000","last_fullsized":"2024-09-10T20:23:43.847367+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6223,"log_dups_size":0,"ondisk_log_size":6223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50651136,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":483,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6130","reported_seq":6197,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.688293+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:45.688293+0000","last_peered":"2024-09-10T20:23:45.688293+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:45.688293+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468576,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6088","reported_seq":6148,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.743827+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:45.743827+0000","last_peered":"2024-09-10T20:23:45.743827+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:45.743827+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6088,"log_dups_size":0,"ondisk_log_size":6088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49233920,"num_objects":6010,"num_object_clones":0,"num_object_copies":24040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6089","reported_seq":6167,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.238669+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:45.238669+0000","last_peered":"2024-09-10T20:23:45.238669+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:45.238669+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993440,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4028}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6198","reported_seq":6279,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.159486+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:45.159486+0000","last_peered":"2024-09-10T20:23:45.159486+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:45.159486+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49927328,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6066","reported_seq":10033,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:45.560170+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:45.560170+0000","last_peered":"2024-09-10T20:23:45.560170+0000","last_clean":"2024-09-10T20:23:45.560170+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:45.560170+0000","last_undegraded":"2024-09-10T20:23:45.560170+0000","last_fullsized":"2024-09-10T20:23:45.560170+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48618064,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6079","reported_seq":7178,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.935101+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:42.935101+0000","last_peered":"2024-09-10T20:23:42.935101+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:42.935101+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49681568,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6119","reported_seq":6188,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.746060+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:45.746060+0000","last_peered":"2024-09-10T20:23:45.746060+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:45.746060+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49098480,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6138","reported_seq":6200,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.729403+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:45.729403+0000","last_peered":"2024-09-10T20:23:45.729403+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:45.729403+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6138,"log_dups_size":0,"ondisk_log_size":6138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49371728,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5920","reported_seq":6008,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.956156+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:42.956156+0000","last_peered":"2024-09-10T20:23:42.956156+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:42.956156+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421456,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6113","reported_seq":6186,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.651419+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:45.651419+0000","last_peered":"2024-09-10T20:23:45.651419+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:45.651419+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993440,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6227","reported_seq":6319,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:41.732110+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:41.732110+0000","last_peered":"2024-09-10T20:23:41.732110+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:41.732110+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6227,"log_dups_size":0,"ondisk_log_size":6227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50967712,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6225,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4260}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6116","reported_seq":6196,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.089153+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:45.089153+0000","last_peered":"2024-09-10T20:23:45.089153+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:45.089153+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6116,"log_dups_size":0,"ondisk_log_size":6116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222816,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6114,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791382184,"num_objects":96609,"num_object_clones":0,"num_object_copies":386436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85098,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1482768384,"data_stored":1482768384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97864,"ondisk_log_size":97864,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742864,"kb_used_data":386304,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93628976,"statfs":{"total":96636764160,"available":95876071424,"internally_reserved":0,"allocated":395575296,"data_stored":395371616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,7,3,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727052,"kb_used_data":389052,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93644788,"statfs":{"total":96636764160,"available":95892262912,"internally_reserved":0,"allocated":398389248,"data_stored":398185568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,1,1,4,0,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555060,"kb_used_data":288740,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93816780,"statfs":{"total":96636764160,"available":96068382720,"internally_reserved":0,"allocated":295669760,"data_stored":295466080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742696,"kb_used_data":386200,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93629144,"statfs":{"total":96636764160,"available":95876243456,"internally_reserved":0,"allocated":395468800,"data_stored":395265120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395108352,"data_stored":395108352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295297024,"data_stored":295297024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397897728,"data_stored":397897728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394465280,"data_stored":394465280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:46.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:47.075 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:47.076 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-10T20:23:45.961092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791382184,"num_objects":96609,"num_object_clones":0,"num_object_copies":386436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85098,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97864,"ondisk_log_size":97864,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767672,"kb_used_data":1450296,"kb_used_omap":14,"kb_used_meta":1317297,"kb_avail":374719688,"statfs":{"total":386547056640,"available":383712960512,"internally_reserved":0,"allocated":1485103104,"data_stored":1484288384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1348912694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,8,4,8,7,0,3,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18178072,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3414,"num_write_kb":22529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003541"},"pg_stats":[{"pgid":"1.f","version":"38'6126","reported_seq":6186,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.693407+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:45.693407+0000","last_peered":"2024-09-10T20:23:45.693407+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:45.693407+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49119232,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6156","reported_seq":6282,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:43.815124+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:43.815124+0000","last_peered":"2024-09-10T20:23:43.815124+0000","last_clean":"2024-09-10T20:23:43.815124+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:43.815124+0000","last_undegraded":"2024-09-10T20:23:43.815124+0000","last_fullsized":"2024-09-10T20:23:43.815124+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6076","reported_seq":6164,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.969766+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:42.969766+0000","last_peered":"2024-09-10T20:23:42.969766+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:42.969766+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49674856,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6075,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6223","reported_seq":10387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:43.847367+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:43.847367+0000","last_peered":"2024-09-10T20:23:43.847367+0000","last_clean":"2024-09-10T20:23:43.847367+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:43.847367+0000","last_undegraded":"2024-09-10T20:23:43.847367+0000","last_fullsized":"2024-09-10T20:23:43.847367+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6223,"log_dups_size":0,"ondisk_log_size":6223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50651136,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":483,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6130","reported_seq":6197,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.688293+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:45.688293+0000","last_peered":"2024-09-10T20:23:45.688293+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:45.688293+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468576,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6088","reported_seq":6148,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.743827+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:45.743827+0000","last_peered":"2024-09-10T20:23:45.743827+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:45.743827+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6088,"log_dups_size":0,"ondisk_log_size":6088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49233920,"num_objects":6010,"num_object_clones":0,"num_object_copies":24040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6089","reported_seq":6167,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.238669+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:45.238669+0000","last_peered":"2024-09-10T20:23:45.238669+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:45.238669+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993440,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4028}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6198","reported_seq":6279,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.159486+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:45.159486+0000","last_peered":"2024-09-10T20:23:45.159486+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:45.159486+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49927328,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6066","reported_seq":10033,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:45.560170+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:45.560170+0000","last_peered":"2024-09-10T20:23:45.560170+0000","last_clean":"2024-09-10T20:23:45.560170+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:45.560170+0000","last_undegraded":"2024-09-10T20:23:45.560170+0000","last_fullsized":"2024-09-10T20:23:45.560170+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48618064,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6079","reported_seq":7178,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.935101+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:42.935101+0000","last_peered":"2024-09-10T20:23:42.935101+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:42.935101+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49681568,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6119","reported_seq":6188,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.746060+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:45.746060+0000","last_peered":"2024-09-10T20:23:45.746060+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:45.746060+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49098480,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6138","reported_seq":6200,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.729403+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:45.729403+0000","last_peered":"2024-09-10T20:23:45.729403+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:45.729403+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6138,"log_dups_size":0,"ondisk_log_size":6138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49371728,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5920","reported_seq":6008,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.956156+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:42.956156+0000","last_peered":"2024-09-10T20:23:42.956156+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:42.956156+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421456,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6113","reported_seq":6186,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.651419+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:45.651419+0000","last_peered":"2024-09-10T20:23:45.651419+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:45.651419+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993440,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6227","reported_seq":6319,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:41.732110+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:41.732110+0000","last_peered":"2024-09-10T20:23:41.732110+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:41.732110+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6227,"log_dups_size":0,"ondisk_log_size":6227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50967712,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6225,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4260}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6116","reported_seq":6196,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.089153+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:45.089153+0000","last_peered":"2024-09-10T20:23:45.089153+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:45.089153+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6116,"log_dups_size":0,"ondisk_log_size":6116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222816,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6114,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791382184,"num_objects":96609,"num_object_clones":0,"num_object_copies":386436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85098,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1482768384,"data_stored":1482768384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97864,"ondisk_log_size":97864,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742864,"kb_used_data":386304,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93628976,"statfs":{"total":96636764160,"available":95876071424,"internally_reserved":0,"allocated":395575296,"data_stored":395371616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,7,3,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727052,"kb_used_data":389052,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93644788,"statfs":{"total":96636764160,"available":95892262912,"internally_reserved":0,"allocated":398389248,"data_stored":398185568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,1,1,4,0,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555060,"kb_used_data":288740,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93816780,"statfs":{"total":96636764160,"available":96068382720,"internally_reserved":0,"allocated":295669760,"data_stored":295466080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742696,"kb_used_data":386200,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93629144,"statfs":{"total":96636764160,"available":95876243456,"internally_reserved":0,"allocated":395468800,"data_stored":395265120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395108352,"data_stored":395108352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295297024,"data_stored":295297024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397897728,"data_stored":397897728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394465280,"data_stored":394465280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6126", 'reported_seq': 6186, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.693407+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:45.693407+0000', 'last_peered': '2024-09-10T20:23:45.693407+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:45.693407+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6126, 'log_dups_size': 0, 'ondisk_log_size': 6126, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49119232, 'num_objects': 5996, 'num_object_clones': 0, 'num_object_copies': 23984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6126, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4033}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6076", 'reported_seq': 6164, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:42.969766+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:42.969766+0000', 'last_peered': '2024-09-10T20:23:42.969766+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:42.969766+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6076, 'log_dups_size': 0, 'ondisk_log_size': 6076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49674856, 'num_objects': 6065, 'num_object_clones': 0, 'num_object_copies': 24260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6065, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6075, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4103}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6130", 'reported_seq': 6197, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.688293+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:45.688293+0000', 'last_peered': '2024-09-10T20:23:45.688293+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:45.688293+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6130, 'log_dups_size': 0, 'ondisk_log_size': 6130, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49468576, 'num_objects': 6039, 'num_object_clones': 0, 'num_object_copies': 24156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6128, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4060}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6088", 'reported_seq': 6148, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.743827+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:45.743827+0000', 'last_peered': '2024-09-10T20:23:45.743827+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:45.743827+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6088, 'log_dups_size': 0, 'ondisk_log_size': 6088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49233920, 'num_objects': 6010, 'num_object_clones': 0, 'num_object_copies': 24040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6088, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4075}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6089", 'reported_seq': 6167, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.238669+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:45.238669+0000', 'last_peered': '2024-09-10T20:23:45.238669+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:45.238669+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6089, 'log_dups_size': 0, 'ondisk_log_size': 6089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48993440, 'num_objects': 5981, 'num_object_clones': 0, 'num_object_copies': 23924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6087, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4028}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6198", 'reported_seq': 6279, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.159486+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:45.159486+0000', 'last_peered': '2024-09-10T20:23:45.159486+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:45.159486+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6198, 'log_dups_size': 0, 'ondisk_log_size': 6198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49927328, 'num_objects': 6095, 'num_object_clones': 0, 'num_object_copies': 24380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6196, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6079", 'reported_seq': 7178, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:42.935101+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:42.935101+0000', 'last_peered': '2024-09-10T20:23:42.935101+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:42.935101+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6079, 'log_dups_size': 0, 'ondisk_log_size': 6079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49681568, 'num_objects': 6065, 'num_object_clones': 0, 'num_object_copies': 24260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6077, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 507, 'num_bytes_recovered': 4136960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4667}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6119", 'reported_seq': 6188, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.746060+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:45.746060+0000', 'last_peered': '2024-09-10T20:23:45.746060+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:45.746060+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6119, 'log_dups_size': 0, 'ondisk_log_size': 6119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49098480, 'num_objects': 5994, 'num_object_clones': 0, 'num_object_copies': 23976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1958, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6116, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6138", 'reported_seq': 6200, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.729403+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:45.729403+0000', 'last_peered': '2024-09-10T20:23:45.729403+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:45.729403+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6138, 'log_dups_size': 0, 'ondisk_log_size': 6138, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49371728, 'num_objects': 6027, 'num_object_clones': 0, 'num_object_copies': 24108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6137, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'5920", 'reported_seq': 6008, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:42.956156+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:42.956156+0000', 'last_peered': '2024-09-10T20:23:42.956156+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:42.956156+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5920, 'log_dups_size': 0, 'ondisk_log_size': 5920, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48421456, 'num_objects': 5911, 'num_object_clones': 0, 'num_object_copies': 23644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5919, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6113", 'reported_seq': 6186, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.651419+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:45.651419+0000', 'last_peered': '2024-09-10T20:23:45.651419+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:45.651419+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6113, 'log_dups_size': 0, 'ondisk_log_size': 6113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48993440, 'num_objects': 5981, 'num_object_clones': 0, 'num_object_copies': 23924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1972, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6111, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:47.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6227", 'reported_seq': 6319, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:41.732110+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:41.732110+0000', 'last_peered': '2024-09-10T20:23:41.732110+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:41.732110+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6227, 'log_dups_size': 0, 'ondisk_log_size': 6227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50967712, 'num_objects': 6222, 'num_object_clones': 0, 'num_object_copies': 24888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6225, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4260}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:47.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:47.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6116", 'reported_seq': 6196, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:45.089153+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:45.089153+0000', 'last_peered': '2024-09-10T20:23:45.089153+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:45.089153+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6116, 'log_dups_size': 0, 'ondisk_log_size': 6116, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49222816, 'num_objects': 6009, 'num_object_clones': 0, 'num_object_copies': 24036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6114, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4032}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:47.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:47.831 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:47.831 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25036/386436 objects degraded (6.479%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":316,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96609,"data_bytes":791382184,"bytes_used":2834096128,"bytes_avail":383712960512,"bytes_total":386547056640,"degraded_objects":25036,"degraded_total":386436,"degraded_ratio":0.064786924613648836,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":201327,"recovering_keys_per_sec":0,"num_objects_recovered":315,"num_bytes_recovered":2416640,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1921907,"read_op_per_sec":0,"write_op_per_sec":284},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:47.857 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:47.857 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:48.504 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-10T20:23:47.961720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791382184,"num_objects":96609,"num_object_clones":0,"num_object_copies":386436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85098,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97864,"ondisk_log_size":97864,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767672,"kb_used_data":1450296,"kb_used_omap":14,"kb_used_meta":1317297,"kb_avail":374719688,"statfs":{"total":386547056640,"available":383712960512,"internally_reserved":0,"allocated":1485103104,"data_stored":1484288384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1348912694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,8,4,8,7,0,3,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10420248,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2467,"num_write_kb":14953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003675"},"pg_stats":[{"pgid":"1.f","version":"38'6126","reported_seq":6186,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.693407+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:45.693407+0000","last_peered":"2024-09-10T20:23:45.693407+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:45.693407+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49119232,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6156","reported_seq":6282,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:43.815124+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:43.815124+0000","last_peered":"2024-09-10T20:23:43.815124+0000","last_clean":"2024-09-10T20:23:43.815124+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:43.815124+0000","last_undegraded":"2024-09-10T20:23:43.815124+0000","last_fullsized":"2024-09-10T20:23:43.815124+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6076","reported_seq":6164,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.969766+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:42.969766+0000","last_peered":"2024-09-10T20:23:42.969766+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:42.969766+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49674856,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6075,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6223","reported_seq":10387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:43.847367+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:43.847367+0000","last_peered":"2024-09-10T20:23:43.847367+0000","last_clean":"2024-09-10T20:23:43.847367+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:43.847367+0000","last_undegraded":"2024-09-10T20:23:43.847367+0000","last_fullsized":"2024-09-10T20:23:43.847367+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6223,"log_dups_size":0,"ondisk_log_size":6223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50651136,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":483,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6130","reported_seq":6197,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.688293+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:45.688293+0000","last_peered":"2024-09-10T20:23:45.688293+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:45.688293+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468576,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6088","reported_seq":6148,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.743827+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:45.743827+0000","last_peered":"2024-09-10T20:23:45.743827+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:45.743827+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6088,"log_dups_size":0,"ondisk_log_size":6088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49233920,"num_objects":6010,"num_object_clones":0,"num_object_copies":24040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6089","reported_seq":6167,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.238669+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:45.238669+0000","last_peered":"2024-09-10T20:23:45.238669+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:45.238669+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993440,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4028}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6198","reported_seq":6279,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.159486+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:45.159486+0000","last_peered":"2024-09-10T20:23:45.159486+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:45.159486+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49927328,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6066","reported_seq":10033,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:45.560170+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:45.560170+0000","last_peered":"2024-09-10T20:23:45.560170+0000","last_clean":"2024-09-10T20:23:45.560170+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:45.560170+0000","last_undegraded":"2024-09-10T20:23:45.560170+0000","last_fullsized":"2024-09-10T20:23:45.560170+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48618064,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6079","reported_seq":7178,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.935101+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:42.935101+0000","last_peered":"2024-09-10T20:23:42.935101+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:42.935101+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49681568,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6119","reported_seq":6188,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.746060+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:45.746060+0000","last_peered":"2024-09-10T20:23:45.746060+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:45.746060+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49098480,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6138","reported_seq":6200,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.729403+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:45.729403+0000","last_peered":"2024-09-10T20:23:45.729403+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:45.729403+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6138,"log_dups_size":0,"ondisk_log_size":6138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49371728,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5920","reported_seq":6008,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:42.956156+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:42.956156+0000","last_peered":"2024-09-10T20:23:42.956156+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:42.956156+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421456,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6113","reported_seq":6186,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.651419+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:45.651419+0000","last_peered":"2024-09-10T20:23:45.651419+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:45.651419+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993440,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6227","reported_seq":6319,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:41.732110+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:41.732110+0000","last_peered":"2024-09-10T20:23:41.732110+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:41.732110+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6227,"log_dups_size":0,"ondisk_log_size":6227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50967712,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6225,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4260}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6116","reported_seq":6196,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:45.089153+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:45.089153+0000","last_peered":"2024-09-10T20:23:45.089153+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:45.089153+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6116,"log_dups_size":0,"ondisk_log_size":6116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222816,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6114,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791382184,"num_objects":96609,"num_object_clones":0,"num_object_copies":386436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85098,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1482768384,"data_stored":1482768384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97864,"ondisk_log_size":97864,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742864,"kb_used_data":386304,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93628976,"statfs":{"total":96636764160,"available":95876071424,"internally_reserved":0,"allocated":395575296,"data_stored":395371616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,7,3,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727052,"kb_used_data":389052,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93644788,"statfs":{"total":96636764160,"available":95892262912,"internally_reserved":0,"allocated":398389248,"data_stored":398185568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,1,1,4,0,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555060,"kb_used_data":288740,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93816780,"statfs":{"total":96636764160,"available":96068382720,"internally_reserved":0,"allocated":295669760,"data_stored":295466080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742696,"kb_used_data":386200,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93629144,"statfs":{"total":96636764160,"available":95876243456,"internally_reserved":0,"allocated":395468800,"data_stored":395265120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395108352,"data_stored":395108352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295297024,"data_stored":295297024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397897728,"data_stored":397897728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394465280,"data_stored":394465280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:51.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:52.140 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:52.141 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-10T20:23:51.962947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":780888232,"num_objects":95328,"num_object_clones":0,"num_object_copies":381312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86379,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99145,"ondisk_log_size":99145,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786492,"kb_used_data":1434364,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374700868,"statfs":{"total":386547056640,"available":383693688832,"internally_reserved":0,"allocated":1468788736,"data_stored":1467974016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1384433206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,0,3,14,4,4,0,2,1,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4251624,"num_objects":-518,"num_object_clones":0,"num_object_copies":-2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3238,"num_write_kb":10873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":2842624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003735"},"pg_stats":[{"pgid":"1.f","version":"38'6194","reported_seq":6266,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.755768+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:50.755768+0000","last_peered":"2024-09-10T20:23:50.755768+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:50.755768+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6245","reported_seq":6372,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:48.890381+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:48.890381+0000","last_peered":"2024-09-10T20:23:48.890381+0000","last_clean":"2024-09-10T20:23:48.890381+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:48.890381+0000","last_undegraded":"2024-09-10T20:23:48.890381+0000","last_fullsized":"2024-09-10T20:23:48.890381+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49209344,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2686,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6160","reported_seq":6254,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:47.904423+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:47.904423+0000","last_peered":"2024-09-10T20:23:47.904423+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:47.904423+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6160,"log_dups_size":0,"ondisk_log_size":6160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48986728,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6159,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6301","reported_seq":10465,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:48.810292+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:48.810292+0000","last_peered":"2024-09-10T20:23:48.810292+0000","last_clean":"2024-09-10T20:23:48.810292+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:48.810292+0000","last_undegraded":"2024-09-10T20:23:48.810292+0000","last_fullsized":"2024-09-10T20:23:48.810292+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50012160,"num_objects":6105,"num_object_clones":0,"num_object_copies":24420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":561,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6215","reported_seq":6292,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.768950+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:50.768950+0000","last_peered":"2024-09-10T20:23:50.768950+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:50.768950+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6215,"log_dups_size":0,"ondisk_log_size":6215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48772256,"num_objects":5954,"num_object_clones":0,"num_object_copies":23816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6169","reported_seq":6237,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.636987+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:50.636987+0000","last_peered":"2024-09-10T20:23:50.636987+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:50.636987+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6169,"log_dups_size":0,"ondisk_log_size":6169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48570368,"num_objects":5929,"num_object_clones":0,"num_object_copies":23716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6174","reported_seq":6255,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.247168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:50.247168+0000","last_peered":"2024-09-10T20:23:50.247168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:50.247168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48297120,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6273","reported_seq":6354,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.052659+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:50.052659+0000","last_peered":"2024-09-10T20:23:50.052659+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:50.052659+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49312928,"num_objects":6020,"num_object_clones":0,"num_object_copies":24080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6271,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6140","reported_seq":10107,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:50.471847+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:50.471847+0000","last_peered":"2024-09-10T20:23:50.471847+0000","last_clean":"2024-09-10T20:23:50.471847+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:50.471847+0000","last_undegraded":"2024-09-10T20:23:50.471847+0000","last_fullsized":"2024-09-10T20:23:50.471847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48011856,"num_objects":5861,"num_object_clones":0,"num_object_copies":23444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6150","reported_seq":7452,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:48.006918+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:48.006918+0000","last_peered":"2024-09-10T20:23:48.006918+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:48.006918+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4697}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6202","reported_seq":6276,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.758386+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:50.758386+0000","last_peered":"2024-09-10T20:23:50.758386+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:50.758386+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48418544,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6229","reported_seq":6299,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.762470+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:50.762470+0000","last_peered":"2024-09-10T20:23:50.762470+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:50.762470+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6229,"log_dups_size":0,"ondisk_log_size":6229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48626256,"num_objects":5936,"num_object_clones":0,"num_object_copies":23744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3965}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5991","reported_seq":6079,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:47.979990+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:47.979990+0000","last_peered":"2024-09-10T20:23:47.979990+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:47.979990+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47839824,"num_objects":5840,"num_object_clones":0,"num_object_copies":23360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3892}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6192","reported_seq":6267,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.800084+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:50.800084+0000","last_peered":"2024-09-10T20:23:50.800084+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:50.800084+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48346272,"num_objects":5902,"num_object_clones":0,"num_object_copies":23608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6190,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6311","reported_seq":6411,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:47.841876+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:47.841876+0000","last_peered":"2024-09-10T20:23:47.841876+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:47.841876+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50279584,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6199","reported_seq":6283,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.175239+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:50.175239+0000","last_peered":"2024-09-10T20:23:50.175239+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:50.175239+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":780888232,"num_objects":95328,"num_object_clones":0,"num_object_copies":381312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86379,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1465892864,"data_stored":1465892864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99145,"ondisk_log_size":99145,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741880,"kb_used_data":381128,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93629960,"statfs":{"total":96636764160,"available":95877079040,"internally_reserved":0,"allocated":390275072,"data_stored":390071392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,6,2,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744544,"kb_used_data":384320,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93627296,"statfs":{"total":96636764160,"available":95874351104,"internally_reserved":0,"allocated":393543680,"data_stored":393340000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558364,"kb_used_data":287884,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93813476,"statfs":{"total":96636764160,"available":96064999424,"internally_reserved":0,"allocated":294793216,"data_stored":294589536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741704,"kb_used_data":381032,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93630136,"statfs":{"total":96636764160,"available":95877259264,"internally_reserved":0,"allocated":390176768,"data_stored":389973088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389849088,"data_stored":389849088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294412288,"data_stored":294412288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392404992,"data_stored":392404992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389226496,"data_stored":389226496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:52.176 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:52.787 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-10T20:23:51.962947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":780888232,"num_objects":95328,"num_object_clones":0,"num_object_copies":381312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86379,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99145,"ondisk_log_size":99145,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786492,"kb_used_data":1434364,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374700868,"statfs":{"total":386547056640,"available":383693688832,"internally_reserved":0,"allocated":1468788736,"data_stored":1467974016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1384433206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,0,3,14,4,4,0,2,1,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4251624,"num_objects":-518,"num_object_clones":0,"num_object_copies":-2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3238,"num_write_kb":10873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":2842624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003735"},"pg_stats":[{"pgid":"1.f","version":"38'6194","reported_seq":6266,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.755768+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:50.755768+0000","last_peered":"2024-09-10T20:23:50.755768+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:50.755768+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6245","reported_seq":6372,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:48.890381+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:48.890381+0000","last_peered":"2024-09-10T20:23:48.890381+0000","last_clean":"2024-09-10T20:23:48.890381+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:48.890381+0000","last_undegraded":"2024-09-10T20:23:48.890381+0000","last_fullsized":"2024-09-10T20:23:48.890381+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49209344,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2686,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6160","reported_seq":6254,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:47.904423+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:47.904423+0000","last_peered":"2024-09-10T20:23:47.904423+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:47.904423+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6160,"log_dups_size":0,"ondisk_log_size":6160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48986728,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6159,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6301","reported_seq":10465,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:48.810292+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:48.810292+0000","last_peered":"2024-09-10T20:23:48.810292+0000","last_clean":"2024-09-10T20:23:48.810292+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:48.810292+0000","last_undegraded":"2024-09-10T20:23:48.810292+0000","last_fullsized":"2024-09-10T20:23:48.810292+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50012160,"num_objects":6105,"num_object_clones":0,"num_object_copies":24420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":561,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6215","reported_seq":6292,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.768950+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:50.768950+0000","last_peered":"2024-09-10T20:23:50.768950+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:50.768950+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6215,"log_dups_size":0,"ondisk_log_size":6215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48772256,"num_objects":5954,"num_object_clones":0,"num_object_copies":23816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6169","reported_seq":6237,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.636987+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:50.636987+0000","last_peered":"2024-09-10T20:23:50.636987+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:50.636987+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6169,"log_dups_size":0,"ondisk_log_size":6169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48570368,"num_objects":5929,"num_object_clones":0,"num_object_copies":23716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6174","reported_seq":6255,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.247168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:50.247168+0000","last_peered":"2024-09-10T20:23:50.247168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:50.247168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48297120,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6273","reported_seq":6354,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.052659+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:50.052659+0000","last_peered":"2024-09-10T20:23:50.052659+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:50.052659+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49312928,"num_objects":6020,"num_object_clones":0,"num_object_copies":24080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6271,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6140","reported_seq":10107,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:50.471847+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:50.471847+0000","last_peered":"2024-09-10T20:23:50.471847+0000","last_clean":"2024-09-10T20:23:50.471847+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:50.471847+0000","last_undegraded":"2024-09-10T20:23:50.471847+0000","last_fullsized":"2024-09-10T20:23:50.471847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48011856,"num_objects":5861,"num_object_clones":0,"num_object_copies":23444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6150","reported_seq":7452,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:48.006918+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:48.006918+0000","last_peered":"2024-09-10T20:23:48.006918+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:48.006918+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4697}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6202","reported_seq":6276,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.758386+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:50.758386+0000","last_peered":"2024-09-10T20:23:50.758386+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:50.758386+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48418544,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6229","reported_seq":6299,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.762470+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:50.762470+0000","last_peered":"2024-09-10T20:23:50.762470+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:50.762470+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6229,"log_dups_size":0,"ondisk_log_size":6229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48626256,"num_objects":5936,"num_object_clones":0,"num_object_copies":23744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3965}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5991","reported_seq":6079,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:47.979990+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:47.979990+0000","last_peered":"2024-09-10T20:23:47.979990+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:47.979990+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47839824,"num_objects":5840,"num_object_clones":0,"num_object_copies":23360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3892}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6192","reported_seq":6267,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.800084+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:50.800084+0000","last_peered":"2024-09-10T20:23:50.800084+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:50.800084+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48346272,"num_objects":5902,"num_object_clones":0,"num_object_copies":23608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6190,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6311","reported_seq":6411,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:47.841876+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:47.841876+0000","last_peered":"2024-09-10T20:23:47.841876+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:47.841876+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50279584,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6199","reported_seq":6283,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.175239+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:50.175239+0000","last_peered":"2024-09-10T20:23:50.175239+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:50.175239+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":780888232,"num_objects":95328,"num_object_clones":0,"num_object_copies":381312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86379,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1465892864,"data_stored":1465892864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99145,"ondisk_log_size":99145,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741880,"kb_used_data":381128,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93629960,"statfs":{"total":96636764160,"available":95877079040,"internally_reserved":0,"allocated":390275072,"data_stored":390071392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,6,2,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744544,"kb_used_data":384320,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93627296,"statfs":{"total":96636764160,"available":95874351104,"internally_reserved":0,"allocated":393543680,"data_stored":393340000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558364,"kb_used_data":287884,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93813476,"statfs":{"total":96636764160,"available":96064999424,"internally_reserved":0,"allocated":294793216,"data_stored":294589536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741704,"kb_used_data":381032,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93630136,"statfs":{"total":96636764160,"available":95877259264,"internally_reserved":0,"allocated":390176768,"data_stored":389973088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389849088,"data_stored":389849088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294412288,"data_stored":294412288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392404992,"data_stored":392404992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389226496,"data_stored":389226496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6194", 'reported_seq': 6266, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.755768+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:50.755768+0000', 'last_peered': '2024-09-10T20:23:50.755768+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:50.755768+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6194, 'log_dups_size': 0, 'ondisk_log_size': 6194, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48562176, 'num_objects': 5928, 'num_object_clones': 0, 'num_object_copies': 23712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6194, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3971}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6160", 'reported_seq': 6254, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:47.904423+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:47.904423+0000', 'last_peered': '2024-09-10T20:23:47.904423+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:47.904423+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6160, 'log_dups_size': 0, 'ondisk_log_size': 6160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48986728, 'num_objects': 5981, 'num_object_clones': 0, 'num_object_copies': 23924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1961, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5981, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6159, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4021}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6215", 'reported_seq': 6292, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.768950+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:50.768950+0000', 'last_peered': '2024-09-10T20:23:50.768950+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:50.768950+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6215, 'log_dups_size': 0, 'ondisk_log_size': 6215, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48772256, 'num_objects': 5954, 'num_object_clones': 0, 'num_object_copies': 23816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6213, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3980}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6169", 'reported_seq': 6237, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.636987+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:50.636987+0000', 'last_peered': '2024-09-10T20:23:50.636987+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:50.636987+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6169, 'log_dups_size': 0, 'ondisk_log_size': 6169, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48570368, 'num_objects': 5929, 'num_object_clones': 0, 'num_object_copies': 23716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6169, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:52.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6174", 'reported_seq': 6255, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.247168+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:50.247168+0000', 'last_peered': '2024-09-10T20:23:50.247168+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:50.247168+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6174, 'log_dups_size': 0, 'ondisk_log_size': 6174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48297120, 'num_objects': 5896, 'num_object_clones': 0, 'num_object_copies': 23584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6172, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3945}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6273", 'reported_seq': 6354, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.052659+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:50.052659+0000', 'last_peered': '2024-09-10T20:23:50.052659+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:50.052659+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6273, 'log_dups_size': 0, 'ondisk_log_size': 6273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49312928, 'num_objects': 6020, 'num_object_clones': 0, 'num_object_copies': 24080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6271, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3979}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6150", 'reported_seq': 7452, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:48.006918+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:48.006918+0000', 'last_peered': '2024-09-10T20:23:48.006918+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:48.006918+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6150, 'log_dups_size': 0, 'ondisk_log_size': 6150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49099936, 'num_objects': 5994, 'num_object_clones': 0, 'num_object_copies': 23976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6148, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 608, 'num_bytes_recovered': 4964352, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4697}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6202", 'reported_seq': 6276, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.758386+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:50.758386+0000', 'last_peered': '2024-09-10T20:23:50.758386+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:50.758386+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6202, 'log_dups_size': 0, 'ondisk_log_size': 6202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48418544, 'num_objects': 5911, 'num_object_clones': 0, 'num_object_copies': 23644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1956, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6199, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3958}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6229", 'reported_seq': 6299, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.762470+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:50.762470+0000', 'last_peered': '2024-09-10T20:23:50.762470+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:50.762470+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6229, 'log_dups_size': 0, 'ondisk_log_size': 6229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48626256, 'num_objects': 5936, 'num_object_clones': 0, 'num_object_copies': 23744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1972, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6228, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3965}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:52.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'5991", 'reported_seq': 6079, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:47.979990+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:47.979990+0000', 'last_peered': '2024-09-10T20:23:47.979990+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:47.979990+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 5991, 'log_dups_size': 0, 'ondisk_log_size': 5991, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47839824, 'num_objects': 5840, 'num_object_clones': 0, 'num_object_copies': 23360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1949, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5990, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:52.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:52.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6192", 'reported_seq': 6267, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.800084+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:50.800084+0000', 'last_peered': '2024-09-10T20:23:50.800084+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:50.800084+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6192, 'log_dups_size': 0, 'ondisk_log_size': 6192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48346272, 'num_objects': 5902, 'num_object_clones': 0, 'num_object_copies': 23608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6190, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:52.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:52.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6311", 'reported_seq': 6411, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:47.841876+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:47.841876+0000', 'last_peered': '2024-09-10T20:23:47.841876+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:47.841876+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6311, 'log_dups_size': 0, 'ondisk_log_size': 6311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50279584, 'num_objects': 6138, 'num_object_clones': 0, 'num_object_copies': 24552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6309, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:52.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:52.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6199", 'reported_seq': 6283, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:50.175239+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:50.175239+0000', 'last_peered': '2024-09-10T20:23:50.175239+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:50.175239+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6199, 'log_dups_size': 0, 'ondisk_log_size': 6199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48542880, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6197, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3951}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:52.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:53.574 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:53.574 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24902/381312 objects degraded (6.531%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":322,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95328,"data_bytes":780888232,"bytes_used":2853367808,"bytes_avail":383693688832,"bytes_total":386547056640,"degraded_objects":24902,"degraded_total":381312,"degraded_ratio":0.065306101040617653,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":236811,"recovering_keys_per_sec":0,"num_objects_recovered":400,"num_bytes_recovered":2842624,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":927540,"read_op_per_sec":0,"write_op_per_sec":269},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:22:39.935879+0000","services":{}},"progress_events":{}} 2024-09-10T20:23:53.607 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:53.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:54.183 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:54.218 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-10T20:23:53.963673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":776825000,"num_objects":94832,"num_object_clones":0,"num_object_copies":379328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86875,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99641,"ondisk_log_size":99641,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788552,"kb_used_data":1428108,"kb_used_omap":14,"kb_used_meta":1360305,"kb_avail":374698808,"statfs":{"total":386547056640,"available":383691579392,"internally_reserved":0,"allocated":1462382592,"data_stored":1461567872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1392952886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,0,14,13,6,5,0,2,0,1,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15114216,"num_objects":-1844,"num_object_clones":0,"num_object_copies":-7376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2904,"num_write_kb":4233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":484,"num_bytes_recovered":3432448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003820"},"pg_stats":[{"pgid":"1.f","version":"38'6194","reported_seq":6266,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.755768+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:50.755768+0000","last_peered":"2024-09-10T20:23:50.755768+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:50.755768+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6313","reported_seq":6440,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.886923+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:53.886923+0000","last_peered":"2024-09-10T20:23:53.886923+0000","last_clean":"2024-09-10T20:23:53.886923+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:53.886923+0000","last_undegraded":"2024-09-10T20:23:53.886923+0000","last_fullsized":"2024-09-10T20:23:53.886923+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48652288,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2754,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6243","reported_seq":6345,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.916444+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:52.916444+0000","last_peered":"2024-09-10T20:23:52.916444+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:52.916444+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48306792,"num_objects":5898,"num_object_clones":0,"num_object_copies":23592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6242,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6390","reported_seq":10554,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.828851+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:53.828851+0000","last_peered":"2024-09-10T20:23:53.828851+0000","last_clean":"2024-09-10T20:23:53.828851+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:53.828851+0000","last_undegraded":"2024-09-10T20:23:53.828851+0000","last_fullsized":"2024-09-10T20:23:53.828851+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49283072,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6215","reported_seq":6292,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.768950+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:50.768950+0000","last_peered":"2024-09-10T20:23:50.768950+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:50.768950+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6215,"log_dups_size":0,"ondisk_log_size":6215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48772256,"num_objects":5954,"num_object_clones":0,"num_object_copies":23816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6169","reported_seq":6237,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.636987+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:50.636987+0000","last_peered":"2024-09-10T20:23:50.636987+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:50.636987+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6169,"log_dups_size":0,"ondisk_log_size":6169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48570368,"num_objects":5929,"num_object_clones":0,"num_object_copies":23716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6174","reported_seq":6255,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.247168+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:50.247168+0000","last_peered":"2024-09-10T20:23:50.247168+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:50.247168+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48297120,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6273","reported_seq":6354,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.052659+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:50.052659+0000","last_peered":"2024-09-10T20:23:50.052659+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:50.052659+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49312928,"num_objects":6020,"num_object_clones":0,"num_object_copies":24080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6271,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6140","reported_seq":10107,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:50.471847+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:50.471847+0000","last_peered":"2024-09-10T20:23:50.471847+0000","last_clean":"2024-09-10T20:23:50.471847+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:50.471847+0000","last_undegraded":"2024-09-10T20:23:50.471847+0000","last_fullsized":"2024-09-10T20:23:50.471847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48011856,"num_objects":5861,"num_object_clones":0,"num_object_copies":23444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6241","reported_seq":7688,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:53.003932+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:53.003932+0000","last_peered":"2024-09-10T20:23:53.003932+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:53.003932+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6241,"log_dups_size":0,"ondisk_log_size":6241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48354464,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6202","reported_seq":6276,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.758386+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:50.758386+0000","last_peered":"2024-09-10T20:23:50.758386+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:50.758386+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48418544,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6229","reported_seq":6299,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.762470+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:50.762470+0000","last_peered":"2024-09-10T20:23:50.762470+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:50.762470+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6229,"log_dups_size":0,"ondisk_log_size":6229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48626256,"num_objects":5936,"num_object_clones":0,"num_object_copies":23744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3965}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6076","reported_seq":6174,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.802953+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:52.802953+0000","last_peered":"2024-09-10T20:23:52.802953+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:52.802953+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143504,"num_objects":5755,"num_object_clones":0,"num_object_copies":23020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6075,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6192","reported_seq":6267,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.800084+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:50.800084+0000","last_peered":"2024-09-10T20:23:50.800084+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:50.800084+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48346272,"num_objects":5902,"num_object_clones":0,"num_object_copies":23608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6190,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6391","reported_seq":6497,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.918157+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:52.918157+0000","last_peered":"2024-09-10T20:23:52.918157+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:52.918157+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49624224,"num_objects":6058,"num_object_clones":0,"num_object_copies":24232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6199","reported_seq":6283,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:50.175239+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:50.175239+0000","last_peered":"2024-09-10T20:23:50.175239+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:50.175239+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":776825000,"num_objects":94832,"num_object_clones":0,"num_object_copies":379328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86875,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1459220480,"data_stored":1459220480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99641,"ondisk_log_size":99641,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741880,"kb_used_data":381128,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93629960,"statfs":{"total":96636764160,"available":95877079040,"internally_reserved":0,"allocated":390275072,"data_stored":390071392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,6,2,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743428,"kb_used_data":379048,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93628412,"statfs":{"total":96636764160,"available":95875493888,"internally_reserved":0,"allocated":388145152,"data_stored":387941472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,5,2,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561540,"kb_used_data":286900,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93810300,"statfs":{"total":96636764160,"available":96061747200,"internally_reserved":0,"allocated":293785600,"data_stored":293581920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741704,"kb_used_data":381032,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93630136,"statfs":{"total":96636764160,"available":95877259264,"internally_reserved":0,"allocated":390176768,"data_stored":389973088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389849088,"data_stored":389849088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293335040,"data_stored":293335040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386809856,"data_stored":386809856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389226496,"data_stored":389226496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:57.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:57.791 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:57.791 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:57.816 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-10T20:23:55.964621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":769632424,"num_objects":93954,"num_object_clones":0,"num_object_copies":375816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87753,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100519,"ondisk_log_size":100519,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785824,"kb_used_data":1417104,"kb_used_omap":14,"kb_used_meta":1368625,"kb_avail":374701536,"statfs":{"total":386547056640,"available":383694372864,"internally_reserved":0,"allocated":1451114496,"data_stored":1450299776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1401472566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,13,15,6,9,6,0,4,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24502272,"num_objects":-2991,"num_object_clones":0,"num_object_copies":-11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3395,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004189"},"pg_stats":[{"pgid":"1.f","version":"38'6292","reported_seq":6382,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.751558+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:55.751558+0000","last_peered":"2024-09-10T20:23:55.751558+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:55.751558+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47759360,"num_objects":5830,"num_object_clones":0,"num_object_copies":23320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6313","reported_seq":6440,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.886923+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:53.886923+0000","last_peered":"2024-09-10T20:23:53.886923+0000","last_clean":"2024-09-10T20:23:53.886923+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:53.886923+0000","last_undegraded":"2024-09-10T20:23:53.886923+0000","last_fullsized":"2024-09-10T20:23:53.886923+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48652288,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2754,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6243","reported_seq":6345,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.916444+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:52.916444+0000","last_peered":"2024-09-10T20:23:52.916444+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:52.916444+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48306792,"num_objects":5898,"num_object_clones":0,"num_object_copies":23592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6242,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6390","reported_seq":10554,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.828851+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:53.828851+0000","last_peered":"2024-09-10T20:23:53.828851+0000","last_clean":"2024-09-10T20:23:53.828851+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:53.828851+0000","last_undegraded":"2024-09-10T20:23:53.828851+0000","last_fullsized":"2024-09-10T20:23:53.828851+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49283072,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6290","reported_seq":6377,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.749603+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:55.749603+0000","last_peered":"2024-09-10T20:23:55.749603+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:55.749603+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6290,"log_dups_size":0,"ondisk_log_size":6290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48157856,"num_objects":5879,"num_object_clones":0,"num_object_copies":23516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6261","reported_seq":6330,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.823625+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:55.823625+0000","last_peered":"2024-09-10T20:23:55.823625+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:55.823625+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47816704,"num_objects":5837,"num_object_clones":0,"num_object_copies":23348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6261,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6262","reported_seq":6344,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.153900+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:55.153900+0000","last_peered":"2024-09-10T20:23:55.153900+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:55.153900+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47576224,"num_objects":5808,"num_object_clones":0,"num_object_copies":23232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3857}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6367","reported_seq":6450,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.229951+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:55.229951+0000","last_peered":"2024-09-10T20:23:55.229951+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:55.229951+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6367,"log_dups_size":0,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6224","reported_seq":10191,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:55.823433+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:55.823433+0000","last_peered":"2024-09-10T20:23:55.823433+0000","last_clean":"2024-09-10T20:23:55.823433+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:55.823433+0000","last_undegraded":"2024-09-10T20:23:55.823433+0000","last_fullsized":"2024-09-10T20:23:55.823433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323728,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2777,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6241","reported_seq":7688,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:53.003932+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:53.003932+0000","last_peered":"2024-09-10T20:23:53.003932+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:53.003932+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6241,"log_dups_size":0,"ondisk_log_size":6241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48354464,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6299","reported_seq":6383,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.750685+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:55.750685+0000","last_peered":"2024-09-10T20:23:55.750685+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:55.750685+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6299,"log_dups_size":0,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47623920,"num_objects":5814,"num_object_clones":0,"num_object_copies":23256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6314","reported_seq":6396,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.852153+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:55.852153+0000","last_peered":"2024-09-10T20:23:55.852153+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:55.852153+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47929936,"num_objects":5851,"num_object_clones":0,"num_object_copies":23404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3886}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6076","reported_seq":6174,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.802953+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:52.802953+0000","last_peered":"2024-09-10T20:23:52.802953+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:52.802953+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143504,"num_objects":5755,"num_object_clones":0,"num_object_copies":23020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6075,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6267","reported_seq":6357,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.767853+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:55.767853+0000","last_peered":"2024-09-10T20:23:55.767853+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:55.767853+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47731872,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3865}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6391","reported_seq":6497,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.918157+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:52.918157+0000","last_peered":"2024-09-10T20:23:52.918157+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:52.918157+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49624224,"num_objects":6058,"num_object_clones":0,"num_object_copies":24232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6289","reported_seq":6376,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.201756+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:55.201756+0000","last_peered":"2024-09-10T20:23:55.201756+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:55.201756+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47805600,"num_objects":5836,"num_object_clones":0,"num_object_copies":23344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":769632424,"num_objects":93954,"num_object_clones":0,"num_object_copies":375816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87753,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1447862272,"data_stored":1447862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100519,"ondisk_log_size":100519,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740492,"kb_used_data":375612,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93631348,"statfs":{"total":96636764160,"available":95878500352,"internally_reserved":0,"allocated":384626688,"data_stored":384423008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,1,0,4,4,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743428,"kb_used_data":379048,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93628412,"statfs":{"total":96636764160,"available":95875493888,"internally_reserved":0,"allocated":388145152,"data_stored":387941472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,5,2,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561540,"kb_used_data":286900,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93810300,"statfs":{"total":96636764160,"available":96061747200,"internally_reserved":0,"allocated":293785600,"data_stored":293581920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740364,"kb_used_data":375544,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93631476,"statfs":{"total":96636764160,"available":95878631424,"internally_reserved":0,"allocated":384557056,"data_stored":384353376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384184320,"data_stored":384184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293335040,"data_stored":293335040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386809856,"data_stored":386809856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383533056,"data_stored":383533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:57.817 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:58.295 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:58.296 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:58.331 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-10T20:23:57.966735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":769632424,"num_objects":93954,"num_object_clones":0,"num_object_copies":375816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87753,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100519,"ondisk_log_size":100519,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785824,"kb_used_data":1417104,"kb_used_omap":14,"kb_used_meta":1368625,"kb_avail":374701536,"statfs":{"total":386547056640,"available":383694372864,"internally_reserved":0,"allocated":1451114496,"data_stored":1450299776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1401472566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,13,15,6,9,6,0,4,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21749760,"num_objects":-2655,"num_object_clones":0,"num_object_copies":-10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005642"},"pg_stats":[{"pgid":"1.f","version":"38'6292","reported_seq":6382,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.751558+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:55.751558+0000","last_peered":"2024-09-10T20:23:55.751558+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:55.751558+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47759360,"num_objects":5830,"num_object_clones":0,"num_object_copies":23320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6313","reported_seq":6440,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.886923+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:53.886923+0000","last_peered":"2024-09-10T20:23:53.886923+0000","last_clean":"2024-09-10T20:23:53.886923+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:53.886923+0000","last_undegraded":"2024-09-10T20:23:53.886923+0000","last_fullsized":"2024-09-10T20:23:53.886923+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48652288,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2754,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6243","reported_seq":6345,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.916444+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:52.916444+0000","last_peered":"2024-09-10T20:23:52.916444+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:52.916444+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48306792,"num_objects":5898,"num_object_clones":0,"num_object_copies":23592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6242,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6390","reported_seq":10554,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.828851+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:53.828851+0000","last_peered":"2024-09-10T20:23:53.828851+0000","last_clean":"2024-09-10T20:23:53.828851+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:53.828851+0000","last_undegraded":"2024-09-10T20:23:53.828851+0000","last_fullsized":"2024-09-10T20:23:53.828851+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49283072,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6290","reported_seq":6377,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.749603+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:55.749603+0000","last_peered":"2024-09-10T20:23:55.749603+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:55.749603+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6290,"log_dups_size":0,"ondisk_log_size":6290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48157856,"num_objects":5879,"num_object_clones":0,"num_object_copies":23516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6261","reported_seq":6330,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.823625+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:55.823625+0000","last_peered":"2024-09-10T20:23:55.823625+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:55.823625+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47816704,"num_objects":5837,"num_object_clones":0,"num_object_copies":23348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6261,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6262","reported_seq":6344,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.153900+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:55.153900+0000","last_peered":"2024-09-10T20:23:55.153900+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:55.153900+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47576224,"num_objects":5808,"num_object_clones":0,"num_object_copies":23232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3857}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6367","reported_seq":6450,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.229951+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:55.229951+0000","last_peered":"2024-09-10T20:23:55.229951+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:55.229951+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6367,"log_dups_size":0,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6224","reported_seq":10191,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:55.823433+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:55.823433+0000","last_peered":"2024-09-10T20:23:55.823433+0000","last_clean":"2024-09-10T20:23:55.823433+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:55.823433+0000","last_undegraded":"2024-09-10T20:23:55.823433+0000","last_fullsized":"2024-09-10T20:23:55.823433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323728,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2777,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6241","reported_seq":7688,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:53.003932+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:53.003932+0000","last_peered":"2024-09-10T20:23:53.003932+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:53.003932+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6241,"log_dups_size":0,"ondisk_log_size":6241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48354464,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6299","reported_seq":6383,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.750685+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:55.750685+0000","last_peered":"2024-09-10T20:23:55.750685+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:55.750685+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6299,"log_dups_size":0,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47623920,"num_objects":5814,"num_object_clones":0,"num_object_copies":23256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6314","reported_seq":6396,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.852153+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:55.852153+0000","last_peered":"2024-09-10T20:23:55.852153+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:55.852153+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47929936,"num_objects":5851,"num_object_clones":0,"num_object_copies":23404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3886}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6076","reported_seq":6174,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.802953+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:52.802953+0000","last_peered":"2024-09-10T20:23:52.802953+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:52.802953+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143504,"num_objects":5755,"num_object_clones":0,"num_object_copies":23020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6075,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6267","reported_seq":6357,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.767853+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:55.767853+0000","last_peered":"2024-09-10T20:23:55.767853+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:55.767853+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47731872,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3865}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6391","reported_seq":6497,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.918157+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:52.918157+0000","last_peered":"2024-09-10T20:23:52.918157+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:52.918157+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49624224,"num_objects":6058,"num_object_clones":0,"num_object_copies":24232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6289","reported_seq":6376,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.201756+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:55.201756+0000","last_peered":"2024-09-10T20:23:55.201756+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:55.201756+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47805600,"num_objects":5836,"num_object_clones":0,"num_object_copies":23344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":769632424,"num_objects":93954,"num_object_clones":0,"num_object_copies":375816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87753,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1447862272,"data_stored":1447862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100519,"ondisk_log_size":100519,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740492,"kb_used_data":375612,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93631348,"statfs":{"total":96636764160,"available":95878500352,"internally_reserved":0,"allocated":384626688,"data_stored":384423008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,1,0,4,4,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743428,"kb_used_data":379048,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93628412,"statfs":{"total":96636764160,"available":95875493888,"internally_reserved":0,"allocated":388145152,"data_stored":387941472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,5,2,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561540,"kb_used_data":286900,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93810300,"statfs":{"total":96636764160,"available":96061747200,"internally_reserved":0,"allocated":293785600,"data_stored":293581920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740364,"kb_used_data":375544,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93631476,"statfs":{"total":96636764160,"available":95878631424,"internally_reserved":0,"allocated":384557056,"data_stored":384353376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384184320,"data_stored":384184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293335040,"data_stored":293335040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386809856,"data_stored":386809856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383533056,"data_stored":383533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6292", 'reported_seq': 6382, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.751558+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:23:55.751558+0000', 'last_peered': '2024-09-10T20:23:55.751558+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:23:55.751558+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6292, 'log_dups_size': 0, 'ondisk_log_size': 6292, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47759360, 'num_objects': 5830, 'num_object_clones': 0, 'num_object_copies': 23320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1948, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6292, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3882}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6243", 'reported_seq': 6345, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:52.916444+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:52.916444+0000', 'last_peered': '2024-09-10T20:23:52.916444+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:52.916444+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6243, 'log_dups_size': 0, 'ondisk_log_size': 6243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48306792, 'num_objects': 5898, 'num_object_clones': 0, 'num_object_copies': 23592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5898, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6242, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3942}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6290", 'reported_seq': 6377, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.749603+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:23:55.749603+0000', 'last_peered': '2024-09-10T20:23:55.749603+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:23:55.749603+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6290, 'log_dups_size': 0, 'ondisk_log_size': 6290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48157856, 'num_objects': 5879, 'num_object_clones': 0, 'num_object_copies': 23516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1971, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6288, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3910}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:23:58.332 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6261", 'reported_seq': 6330, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.823625+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:23:55.823625+0000', 'last_peered': '2024-09-10T20:23:55.823625+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:23:55.823625+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6261, 'log_dups_size': 0, 'ondisk_log_size': 6261, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47816704, 'num_objects': 5837, 'num_object_clones': 0, 'num_object_copies': 23348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1930, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6261, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3907}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6262", 'reported_seq': 6344, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.153900+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:23:55.153900+0000', 'last_peered': '2024-09-10T20:23:55.153900+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:23:55.153900+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6262, 'log_dups_size': 0, 'ondisk_log_size': 6262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47576224, 'num_objects': 5808, 'num_object_clones': 0, 'num_object_copies': 23232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6260, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3857}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6367", 'reported_seq': 6450, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.229951+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:23:55.229951+0000', 'last_peered': '2024-09-10T20:23:55.229951+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:23:55.229951+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6367, 'log_dups_size': 0, 'ondisk_log_size': 6367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48542880, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2042, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6365, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6241", 'reported_seq': 7688, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:53.003932+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:53.003932+0000', 'last_peered': '2024-09-10T20:23:53.003932+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:53.003932+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6241, 'log_dups_size': 0, 'ondisk_log_size': 6241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48354464, 'num_objects': 5903, 'num_object_clones': 0, 'num_object_copies': 23612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6239, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 680, 'num_bytes_recovered': 5554176, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4678}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6299", 'reported_seq': 6383, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.750685+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:23:55.750685+0000', 'last_peered': '2024-09-10T20:23:55.750685+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:23:55.750685+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6299, 'log_dups_size': 0, 'ondisk_log_size': 6299, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47623920, 'num_objects': 5814, 'num_object_clones': 0, 'num_object_copies': 23256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1951, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6296, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3866}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:23:58.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6314", 'reported_seq': 6396, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.852153+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:23:55.852153+0000', 'last_peered': '2024-09-10T20:23:55.852153+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:23:55.852153+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6314, 'log_dups_size': 0, 'ondisk_log_size': 6314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47929936, 'num_objects': 5851, 'num_object_clones': 0, 'num_object_copies': 23404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6313, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6076", 'reported_seq': 6174, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:52.802953+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:52.802953+0000', 'last_peered': '2024-09-10T20:23:52.802953+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:52.802953+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6076, 'log_dups_size': 0, 'ondisk_log_size': 6076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47143504, 'num_objects': 5755, 'num_object_clones': 0, 'num_object_copies': 23020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1944, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6075, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3812}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6267", 'reported_seq': 6357, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.767853+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:23:55.767853+0000', 'last_peered': '2024-09-10T20:23:55.767853+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:23:55.767853+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6267, 'log_dups_size': 0, 'ondisk_log_size': 6267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47731872, 'num_objects': 5827, 'num_object_clones': 0, 'num_object_copies': 23308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6265, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3865}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6391", 'reported_seq': 6497, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:52.918157+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:52.918157+0000', 'last_peered': '2024-09-10T20:23:52.918157+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:52.918157+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6391, 'log_dups_size': 0, 'ondisk_log_size': 6391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49624224, 'num_objects': 6058, 'num_object_clones': 0, 'num_object_copies': 24232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6389, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:23:58.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6289", 'reported_seq': 6376, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:55.201756+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:23:55.201756+0000', 'last_peered': '2024-09-10T20:23:55.201756+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:23:55.201756+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6289, 'log_dups_size': 0, 'ondisk_log_size': 6289, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47805600, 'num_objects': 5836, 'num_object_clones': 0, 'num_object_copies': 23344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6287, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3863}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:23:58.335 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:23:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:59.040 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24783/375816 objects degraded (6.594%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":327,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93954,"data_bytes":769632424,"bytes_used":2852683776,"bytes_avail":383694372864,"bytes_total":386547056640,"degraded_objects":24783,"degraded_total":375816,"degraded_ratio":0.065944504757647363,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":118045,"recovering_keys_per_sec":0,"num_objects_recovered":253,"num_bytes_recovered":1417216,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-10T20:23:53.963775+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:23:59.062 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:23:59.062 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:23:59.681 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:23:59.681 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:23:59.707 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-10T20:23:57.966735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":769632424,"num_objects":93954,"num_object_clones":0,"num_object_copies":375816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87753,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100519,"ondisk_log_size":100519,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785824,"kb_used_data":1417104,"kb_used_omap":14,"kb_used_meta":1368625,"kb_avail":374701536,"statfs":{"total":386547056640,"available":383694372864,"internally_reserved":0,"allocated":1451114496,"data_stored":1450299776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1401472566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,13,15,6,9,6,0,4,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21749760,"num_objects":-2655,"num_object_clones":0,"num_object_copies":-10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005642"},"pg_stats":[{"pgid":"1.f","version":"38'6292","reported_seq":6382,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.751558+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:23:55.751558+0000","last_peered":"2024-09-10T20:23:55.751558+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:23:55.751558+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47759360,"num_objects":5830,"num_object_clones":0,"num_object_copies":23320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6313","reported_seq":6440,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.886923+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:53.886923+0000","last_peered":"2024-09-10T20:23:53.886923+0000","last_clean":"2024-09-10T20:23:53.886923+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:53.886923+0000","last_undegraded":"2024-09-10T20:23:53.886923+0000","last_fullsized":"2024-09-10T20:23:53.886923+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48652288,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2754,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6243","reported_seq":6345,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.916444+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:52.916444+0000","last_peered":"2024-09-10T20:23:52.916444+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:52.916444+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48306792,"num_objects":5898,"num_object_clones":0,"num_object_copies":23592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6242,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6390","reported_seq":10554,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:53.828851+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:53.828851+0000","last_peered":"2024-09-10T20:23:53.828851+0000","last_clean":"2024-09-10T20:23:53.828851+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:53.828851+0000","last_undegraded":"2024-09-10T20:23:53.828851+0000","last_fullsized":"2024-09-10T20:23:53.828851+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49283072,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6290","reported_seq":6377,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.749603+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:23:55.749603+0000","last_peered":"2024-09-10T20:23:55.749603+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:23:55.749603+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6290,"log_dups_size":0,"ondisk_log_size":6290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48157856,"num_objects":5879,"num_object_clones":0,"num_object_copies":23516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6261","reported_seq":6330,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.823625+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:23:55.823625+0000","last_peered":"2024-09-10T20:23:55.823625+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:23:55.823625+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47816704,"num_objects":5837,"num_object_clones":0,"num_object_copies":23348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6261,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6262","reported_seq":6344,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.153900+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:23:55.153900+0000","last_peered":"2024-09-10T20:23:55.153900+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:23:55.153900+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47576224,"num_objects":5808,"num_object_clones":0,"num_object_copies":23232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3857}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6367","reported_seq":6450,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.229951+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:23:55.229951+0000","last_peered":"2024-09-10T20:23:55.229951+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:23:55.229951+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6367,"log_dups_size":0,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6224","reported_seq":10191,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:55.823433+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:23:55.823433+0000","last_peered":"2024-09-10T20:23:55.823433+0000","last_clean":"2024-09-10T20:23:55.823433+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:23:55.823433+0000","last_undegraded":"2024-09-10T20:23:55.823433+0000","last_fullsized":"2024-09-10T20:23:55.823433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323728,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2777,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6241","reported_seq":7688,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:53.003932+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:53.003932+0000","last_peered":"2024-09-10T20:23:53.003932+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:53.003932+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6241,"log_dups_size":0,"ondisk_log_size":6241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48354464,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6299","reported_seq":6383,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.750685+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:23:55.750685+0000","last_peered":"2024-09-10T20:23:55.750685+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:23:55.750685+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6299,"log_dups_size":0,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47623920,"num_objects":5814,"num_object_clones":0,"num_object_copies":23256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6314","reported_seq":6396,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.852153+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:23:55.852153+0000","last_peered":"2024-09-10T20:23:55.852153+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:23:55.852153+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47929936,"num_objects":5851,"num_object_clones":0,"num_object_copies":23404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3886}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6076","reported_seq":6174,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.802953+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:52.802953+0000","last_peered":"2024-09-10T20:23:52.802953+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:52.802953+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143504,"num_objects":5755,"num_object_clones":0,"num_object_copies":23020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6075,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6267","reported_seq":6357,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.767853+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:23:55.767853+0000","last_peered":"2024-09-10T20:23:55.767853+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:23:55.767853+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47731872,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3865}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6391","reported_seq":6497,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:52.918157+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:52.918157+0000","last_peered":"2024-09-10T20:23:52.918157+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:52.918157+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49624224,"num_objects":6058,"num_object_clones":0,"num_object_copies":24232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6289","reported_seq":6376,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:55.201756+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:23:55.201756+0000","last_peered":"2024-09-10T20:23:55.201756+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:23:55.201756+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47805600,"num_objects":5836,"num_object_clones":0,"num_object_copies":23344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":769632424,"num_objects":93954,"num_object_clones":0,"num_object_copies":375816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87753,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1447862272,"data_stored":1447862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100519,"ondisk_log_size":100519,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740492,"kb_used_data":375612,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93631348,"statfs":{"total":96636764160,"available":95878500352,"internally_reserved":0,"allocated":384626688,"data_stored":384423008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,1,0,4,4,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743428,"kb_used_data":379048,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93628412,"statfs":{"total":96636764160,"available":95875493888,"internally_reserved":0,"allocated":388145152,"data_stored":387941472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,5,2,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561540,"kb_used_data":286900,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93810300,"statfs":{"total":96636764160,"available":96061747200,"internally_reserved":0,"allocated":293785600,"data_stored":293581920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740364,"kb_used_data":375544,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93631476,"statfs":{"total":96636764160,"available":95878631424,"internally_reserved":0,"allocated":384557056,"data_stored":384353376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384184320,"data_stored":384184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293335040,"data_stored":293335040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386809856,"data_stored":386809856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383533056,"data_stored":383533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:02.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:03.304 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:03.305 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:03.338 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-10T20:24:01.968214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":758810792,"num_objects":92633,"num_object_clones":0,"num_object_copies":370532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89074,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":6307840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101840,"ondisk_log_size":101840,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722956,"kb_used_data":1399996,"kb_used_omap":14,"kb_used_meta":1322865,"kb_avail":374764404,"statfs":{"total":386547056640,"available":383758749696,"internally_reserved":0,"allocated":1433595904,"data_stored":1432781184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1354614326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,10,11,8,6,1,0,0,4,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28663808,"num_objects":-3499,"num_object_clones":0,"num_object_copies":-13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005837"},"pg_stats":[{"pgid":"1.f","version":"38'6376","reported_seq":6473,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.856215+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:00.856215+0000","last_peered":"2024-09-10T20:24:00.856215+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:00.856215+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47071232,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6376,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6395","reported_seq":6522,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:58.778424+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:58.778424+0000","last_peered":"2024-09-10T20:23:58.778424+0000","last_clean":"2024-09-10T20:23:58.778424+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:58.778424+0000","last_undegraded":"2024-09-10T20:23:58.778424+0000","last_fullsized":"2024-09-10T20:23:58.778424+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6395,"log_dups_size":0,"ondisk_log_size":6395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47980544,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6318","reported_seq":6420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.684214+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:57.684214+0000","last_peered":"2024-09-10T20:23:57.684214+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:57.684214+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47692392,"num_objects":5823,"num_object_clones":0,"num_object_copies":23292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6317,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3867}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6486","reported_seq":10650,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:58.830209+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:58.830209+0000","last_peered":"2024-09-10T20:23:58.830209+0000","last_clean":"2024-09-10T20:23:58.830209+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:58.830209+0000","last_undegraded":"2024-09-10T20:23:58.830209+0000","last_fullsized":"2024-09-10T20:23:58.830209+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6379","reported_seq":6484,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.886123+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:00.886123+0000","last_peered":"2024-09-10T20:24:00.886123+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:00.886123+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47428768,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6341","reported_seq":6421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.897396+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:00.897396+0000","last_peered":"2024-09-10T20:24:00.897396+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:00.897396+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47161344,"num_objects":5757,"num_object_clones":0,"num_object_copies":23028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6341,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3832}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6337","reported_seq":6421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.254570+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:00.254570+0000","last_peered":"2024-09-10T20:24:00.254570+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:00.254570+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961824,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6335,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3783}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6450","reported_seq":6531,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.190277+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:00.190277+0000","last_peered":"2024-09-10T20:24:00.190277+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:00.190277+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862944,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6310","reported_seq":10277,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:00.669420+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:00.669420+0000","last_peered":"2024-09-10T20:24:00.669420+0000","last_clean":"2024-09-10T20:24:00.669420+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:00.669420+0000","last_undegraded":"2024-09-10T20:24:00.669420+0000","last_fullsized":"2024-09-10T20:24:00.669420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46619216,"num_objects":5691,"num_object_clones":0,"num_object_copies":22764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6322","reported_seq":7953,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.851231+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:57.851231+0000","last_peered":"2024-09-10T20:23:57.851231+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:57.851231+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47690912,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":772,"num_bytes_recovered":6307840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6382","reported_seq":6472,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.675454+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:00.675454+0000","last_peered":"2024-09-10T20:24:00.675454+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:00.675454+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46943984,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6379,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6390","reported_seq":6485,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.612168+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:00.612168+0000","last_peered":"2024-09-10T20:24:00.612168+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:00.612168+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47307344,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3816}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6163","reported_seq":6261,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.622241+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:57.622241+0000","last_peered":"2024-09-10T20:23:57.622241+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:57.622241+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46430800,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3725}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6357","reported_seq":6454,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.891850+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:00.891850+0000","last_peered":"2024-09-10T20:24:00.891850+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:00.891850+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46994592,"num_objects":5737,"num_object_clones":0,"num_object_copies":22948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6355,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6469","reported_seq":6577,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.796936+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:57.796936+0000","last_peered":"2024-09-10T20:23:57.796936+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:57.796936+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985248,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4026}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6365","reported_seq":6454,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.106155+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:00.106155+0000","last_peered":"2024-09-10T20:24:00.106155+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:00.106155+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47183008,"num_objects":5760,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3787}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":758810792,"num_objects":92633,"num_object_clones":0,"num_object_copies":370532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89074,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":6307840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1430544384,"data_stored":1430544384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101840,"ondisk_log_size":101840,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722676,"kb_used_data":370276,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93649164,"statfs":{"total":96636764160,"available":95896743936,"internally_reserved":0,"allocated":379162624,"data_stored":378958944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,3,1,4,1,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725372,"kb_used_data":373476,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93646468,"statfs":{"total":96636764160,"available":95893983232,"internally_reserved":0,"allocated":382439424,"data_stored":382235744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552156,"kb_used_data":285836,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93819684,"statfs":{"total":96636764160,"available":96071356416,"internally_reserved":0,"allocated":292696064,"data_stored":292492384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722752,"kb_used_data":370408,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93649088,"statfs":{"total":96636764160,"available":95896666112,"internally_reserved":0,"allocated":379297792,"data_stored":379094112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378847232,"data_stored":378847232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292261888,"data_stored":292261888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381292544,"data_stored":381292544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378142720,"data_stored":378142720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:03.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:03.947 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:03.976 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-10T20:24:01.968214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":758810792,"num_objects":92633,"num_object_clones":0,"num_object_copies":370532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89074,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":6307840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101840,"ondisk_log_size":101840,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722956,"kb_used_data":1399996,"kb_used_omap":14,"kb_used_meta":1322865,"kb_avail":374764404,"statfs":{"total":386547056640,"available":383758749696,"internally_reserved":0,"allocated":1433595904,"data_stored":1432781184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1354614326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,10,11,8,6,1,0,0,4,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28663808,"num_objects":-3499,"num_object_clones":0,"num_object_copies":-13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005837"},"pg_stats":[{"pgid":"1.f","version":"38'6376","reported_seq":6473,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.856215+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:00.856215+0000","last_peered":"2024-09-10T20:24:00.856215+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:00.856215+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47071232,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6376,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6395","reported_seq":6522,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:58.778424+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:58.778424+0000","last_peered":"2024-09-10T20:23:58.778424+0000","last_clean":"2024-09-10T20:23:58.778424+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:58.778424+0000","last_undegraded":"2024-09-10T20:23:58.778424+0000","last_fullsized":"2024-09-10T20:23:58.778424+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6395,"log_dups_size":0,"ondisk_log_size":6395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47980544,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6318","reported_seq":6420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.684214+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:23:57.684214+0000","last_peered":"2024-09-10T20:23:57.684214+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:23:57.684214+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47692392,"num_objects":5823,"num_object_clones":0,"num_object_copies":23292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6317,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3867}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6486","reported_seq":10650,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:58.830209+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:58.830209+0000","last_peered":"2024-09-10T20:23:58.830209+0000","last_clean":"2024-09-10T20:23:58.830209+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:58.830209+0000","last_undegraded":"2024-09-10T20:23:58.830209+0000","last_fullsized":"2024-09-10T20:23:58.830209+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6379","reported_seq":6484,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.886123+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:00.886123+0000","last_peered":"2024-09-10T20:24:00.886123+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:00.886123+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47428768,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6341","reported_seq":6421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.897396+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:00.897396+0000","last_peered":"2024-09-10T20:24:00.897396+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:00.897396+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47161344,"num_objects":5757,"num_object_clones":0,"num_object_copies":23028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6341,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3832}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6337","reported_seq":6421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.254570+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:00.254570+0000","last_peered":"2024-09-10T20:24:00.254570+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:00.254570+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961824,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6335,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3783}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6450","reported_seq":6531,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.190277+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:00.190277+0000","last_peered":"2024-09-10T20:24:00.190277+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:00.190277+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862944,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6310","reported_seq":10277,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:00.669420+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:00.669420+0000","last_peered":"2024-09-10T20:24:00.669420+0000","last_clean":"2024-09-10T20:24:00.669420+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:00.669420+0000","last_undegraded":"2024-09-10T20:24:00.669420+0000","last_fullsized":"2024-09-10T20:24:00.669420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46619216,"num_objects":5691,"num_object_clones":0,"num_object_copies":22764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6322","reported_seq":7953,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.851231+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:23:57.851231+0000","last_peered":"2024-09-10T20:23:57.851231+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:23:57.851231+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47690912,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":772,"num_bytes_recovered":6307840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6382","reported_seq":6472,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.675454+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:00.675454+0000","last_peered":"2024-09-10T20:24:00.675454+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:00.675454+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46943984,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6379,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6390","reported_seq":6485,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.612168+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:00.612168+0000","last_peered":"2024-09-10T20:24:00.612168+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:00.612168+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47307344,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3816}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6163","reported_seq":6261,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.622241+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:23:57.622241+0000","last_peered":"2024-09-10T20:23:57.622241+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:23:57.622241+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46430800,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3725}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6357","reported_seq":6454,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.891850+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:00.891850+0000","last_peered":"2024-09-10T20:24:00.891850+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:00.891850+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46994592,"num_objects":5737,"num_object_clones":0,"num_object_copies":22948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6355,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6469","reported_seq":6577,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:23:57.796936+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:23:57.796936+0000","last_peered":"2024-09-10T20:23:57.796936+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:23:57.796936+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985248,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4026}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6365","reported_seq":6454,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.106155+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:00.106155+0000","last_peered":"2024-09-10T20:24:00.106155+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:00.106155+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47183008,"num_objects":5760,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3787}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":758810792,"num_objects":92633,"num_object_clones":0,"num_object_copies":370532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89074,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":6307840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1430544384,"data_stored":1430544384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101840,"ondisk_log_size":101840,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722676,"kb_used_data":370276,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93649164,"statfs":{"total":96636764160,"available":95896743936,"internally_reserved":0,"allocated":379162624,"data_stored":378958944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,3,1,4,1,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725372,"kb_used_data":373476,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93646468,"statfs":{"total":96636764160,"available":95893983232,"internally_reserved":0,"allocated":382439424,"data_stored":382235744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552156,"kb_used_data":285836,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93819684,"statfs":{"total":96636764160,"available":96071356416,"internally_reserved":0,"allocated":292696064,"data_stored":292492384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722752,"kb_used_data":370408,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93649088,"statfs":{"total":96636764160,"available":95896666112,"internally_reserved":0,"allocated":379297792,"data_stored":379094112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378847232,"data_stored":378847232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292261888,"data_stored":292261888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381292544,"data_stored":381292544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378142720,"data_stored":378142720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:03.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:03.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6376", 'reported_seq': 6473, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.856215+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:00.856215+0000', 'last_peered': '2024-09-10T20:24:00.856215+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:00.856215+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6376, 'log_dups_size': 0, 'ondisk_log_size': 6376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47071232, 'num_objects': 5746, 'num_object_clones': 0, 'num_object_copies': 22984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1944, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6376, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3802}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:03.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:03.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6318", 'reported_seq': 6420, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:57.684214+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:23:57.684214+0000', 'last_peered': '2024-09-10T20:23:57.684214+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:23:57.684214+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6318, 'log_dups_size': 0, 'ondisk_log_size': 6318, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47692392, 'num_objects': 5823, 'num_object_clones': 0, 'num_object_copies': 23292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5823, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6317, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3867}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:03.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:03.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6379", 'reported_seq': 6484, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.886123+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:00.886123+0000', 'last_peered': '2024-09-10T20:24:00.886123+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:00.886123+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6379, 'log_dups_size': 0, 'ondisk_log_size': 6379, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47428768, 'num_objects': 5790, 'num_object_clones': 0, 'num_object_copies': 23160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1962, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6377, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3830}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6341", 'reported_seq': 6421, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.897396+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:00.897396+0000', 'last_peered': '2024-09-10T20:24:00.897396+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:00.897396+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6341, 'log_dups_size': 0, 'ondisk_log_size': 6341, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47161344, 'num_objects': 5757, 'num_object_clones': 0, 'num_object_copies': 23028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1925, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6341, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3832}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6337", 'reported_seq': 6421, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.254570+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:00.254570+0000', 'last_peered': '2024-09-10T20:24:00.254570+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:00.254570+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6337, 'log_dups_size': 0, 'ondisk_log_size': 6337, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46961824, 'num_objects': 5733, 'num_object_clones': 0, 'num_object_copies': 22932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1952, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6335, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3783}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6450", 'reported_seq': 6531, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.190277+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:00.190277+0000', 'last_peered': '2024-09-10T20:24:00.190277+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:00.190277+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6450, 'log_dups_size': 0, 'ondisk_log_size': 6450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47862944, 'num_objects': 5843, 'num_object_clones': 0, 'num_object_copies': 23372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2042, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6448, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6322", 'reported_seq': 7953, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:57.851231+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:23:57.851231+0000', 'last_peered': '2024-09-10T20:23:57.851231+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:23:57.851231+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6322, 'log_dups_size': 0, 'ondisk_log_size': 6322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47690912, 'num_objects': 5822, 'num_object_clones': 0, 'num_object_copies': 23288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6320, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 772, 'num_bytes_recovered': 6307840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4689}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:03.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6382", 'reported_seq': 6472, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.675454+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:00.675454+0000', 'last_peered': '2024-09-10T20:24:00.675454+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:00.675454+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6382, 'log_dups_size': 0, 'ondisk_log_size': 6382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46943984, 'num_objects': 5731, 'num_object_clones': 0, 'num_object_copies': 22924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1948, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6379, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3786}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6390", 'reported_seq': 6485, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.612168+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:00.612168+0000', 'last_peered': '2024-09-10T20:24:00.612168+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:00.612168+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6390, 'log_dups_size': 0, 'ondisk_log_size': 6390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47307344, 'num_objects': 5775, 'num_object_clones': 0, 'num_object_copies': 23100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6389, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3816}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6163", 'reported_seq': 6261, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:57.622241+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:23:57.622241+0000', 'last_peered': '2024-09-10T20:23:57.622241+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:23:57.622241+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6163, 'log_dups_size': 0, 'ondisk_log_size': 6163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46430800, 'num_objects': 5668, 'num_object_clones': 0, 'num_object_copies': 22672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1944, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6162, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3725}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6357", 'reported_seq': 6454, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.891850+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:00.891850+0000', 'last_peered': '2024-09-10T20:24:00.891850+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:00.891850+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6357, 'log_dups_size': 0, 'ondisk_log_size': 6357, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46994592, 'num_objects': 5737, 'num_object_clones': 0, 'num_object_copies': 22948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6355, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6469", 'reported_seq': 6577, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:23:57.796936+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:23:57.796936+0000', 'last_peered': '2024-09-10T20:23:57.796936+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:23:57.796936+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6469, 'log_dups_size': 0, 'ondisk_log_size': 6469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48985248, 'num_objects': 5980, 'num_object_clones': 0, 'num_object_copies': 23920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1956, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6467, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4026}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:03.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:03.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6365", 'reported_seq': 6454, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:00.106155+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:00.106155+0000', 'last_peered': '2024-09-10T20:24:00.106155+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:00.106155+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6365, 'log_dups_size': 0, 'ondisk_log_size': 6365, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47183008, 'num_objects': 5760, 'num_object_clones': 0, 'num_object_copies': 23040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6363, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3787}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:03.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:04.671 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:04.672 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24588/369180 objects degraded (6.660%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":333,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92295,"data_bytes":756041896,"bytes_used":2786963456,"bytes_avail":383760093184,"bytes_total":386547056640,"degraded_objects":24588,"degraded_total":369180,"degraded_ratio":0.066601657727937594,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":153525,"recovering_keys_per_sec":0,"num_objects_recovered":314,"num_bytes_recovered":1843200,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":252},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-10T20:23:53.963775+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:04.706 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:04.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:05.315 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:05.315 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:05.334 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-10T20:24:03.968732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":756041896,"num_objects":92295,"num_object_clones":0,"num_object_copies":369180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92295,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89412,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102178,"ondisk_log_size":102178,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2721644,"kb_used_data":1394520,"kb_used_omap":14,"kb_used_meta":1327025,"kb_avail":374765716,"statfs":{"total":386547056640,"available":383760093184,"internally_reserved":0,"allocated":1427988480,"data_stored":1427173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1358874166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,2,10,9,9,6,1,0,1,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24846336,"num_objects":-3033,"num_object_clones":0,"num_object_copies":-12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3033,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":1843200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005784"},"pg_stats":[{"pgid":"1.f","version":"38'6376","reported_seq":6473,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.856215+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:00.856215+0000","last_peered":"2024-09-10T20:24:00.856215+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:00.856215+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47071232,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6376,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6395","reported_seq":6522,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:58.778424+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:23:58.778424+0000","last_peered":"2024-09-10T20:23:58.778424+0000","last_clean":"2024-09-10T20:23:58.778424+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:23:58.778424+0000","last_undegraded":"2024-09-10T20:23:58.778424+0000","last_fullsized":"2024-09-10T20:23:58.778424+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6395,"log_dups_size":0,"ondisk_log_size":6395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47980544,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6399","reported_seq":6505,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.768171+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:02.768171+0000","last_peered":"2024-09-10T20:24:02.768171+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:02.768171+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47028840,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6398,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3788}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6486","reported_seq":10650,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:23:58.830209+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:23:58.830209+0000","last_peered":"2024-09-10T20:23:58.830209+0000","last_clean":"2024-09-10T20:23:58.830209+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:23:58.830209+0000","last_undegraded":"2024-09-10T20:23:58.830209+0000","last_fullsized":"2024-09-10T20:23:58.830209+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6379","reported_seq":6484,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.886123+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:00.886123+0000","last_peered":"2024-09-10T20:24:00.886123+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:00.886123+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47428768,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6341","reported_seq":6421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.897396+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:00.897396+0000","last_peered":"2024-09-10T20:24:00.897396+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:00.897396+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47161344,"num_objects":5757,"num_object_clones":0,"num_object_copies":23028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6341,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3832}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6337","reported_seq":6421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.254570+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:00.254570+0000","last_peered":"2024-09-10T20:24:00.254570+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:00.254570+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961824,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6335,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3783}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6450","reported_seq":6531,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.190277+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:00.190277+0000","last_peered":"2024-09-10T20:24:00.190277+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:00.190277+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862944,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6310","reported_seq":10277,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:00.669420+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:00.669420+0000","last_peered":"2024-09-10T20:24:00.669420+0000","last_clean":"2024-09-10T20:24:00.669420+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:00.669420+0000","last_undegraded":"2024-09-10T20:24:00.669420+0000","last_fullsized":"2024-09-10T20:24:00.669420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46619216,"num_objects":5691,"num_object_clones":0,"num_object_copies":22764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6406","reported_seq":8159,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.990816+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:02.990816+0000","last_peered":"2024-09-10T20:24:02.990816+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:02.990816+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47002784,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6382","reported_seq":6472,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.675454+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:00.675454+0000","last_peered":"2024-09-10T20:24:00.675454+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:00.675454+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46943984,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6379,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6390","reported_seq":6485,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.612168+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:00.612168+0000","last_peered":"2024-09-10T20:24:00.612168+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:00.612168+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47307344,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3816}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6252","reported_seq":6360,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.796004+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:02.796004+0000","last_peered":"2024-09-10T20:24:02.796004+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:02.796004+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6252,"log_dups_size":0,"ondisk_log_size":6252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45701712,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6357","reported_seq":6454,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.891850+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:00.891850+0000","last_peered":"2024-09-10T20:24:00.891850+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:00.891850+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46994592,"num_objects":5737,"num_object_clones":0,"num_object_copies":22948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6355,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6553","reported_seq":6665,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.935571+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:02.935571+0000","last_peered":"2024-09-10T20:24:02.935571+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:02.935571+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48297120,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3944}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6365","reported_seq":6454,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:00.106155+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:00.106155+0000","last_peered":"2024-09-10T20:24:00.106155+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:00.106155+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47183008,"num_objects":5760,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3787}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":756041896,"num_objects":92295,"num_object_clones":0,"num_object_copies":369180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92295,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89412,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1424924672,"data_stored":1424924672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102178,"ondisk_log_size":102178,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722676,"kb_used_data":370276,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93649164,"statfs":{"total":96636764160,"available":95896743936,"internally_reserved":0,"allocated":379162624,"data_stored":378958944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,3,1,4,1,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724060,"kb_used_data":368000,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93647780,"statfs":{"total":96636764160,"available":95895326720,"internally_reserved":0,"allocated":376832000,"data_stored":376628320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,4,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552156,"kb_used_data":285836,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93819684,"statfs":{"total":96636764160,"available":96071356416,"internally_reserved":0,"allocated":292696064,"data_stored":292492384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722752,"kb_used_data":370408,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93649088,"statfs":{"total":96636764160,"available":95896666112,"internally_reserved":0,"allocated":379297792,"data_stored":379094112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378847232,"data_stored":378847232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292261888,"data_stored":292261888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375672832,"data_stored":375672832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378142720,"data_stored":378142720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:08.336 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:08.938 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:08.969 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-10T20:24:07.970306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":747743400,"num_objects":91282,"num_object_clones":0,"num_object_copies":365128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90425,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103191,"ondisk_log_size":103191,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730380,"kb_used_data":1382448,"kb_used_omap":14,"kb_used_meta":1347825,"kb_avail":374756980,"statfs":{"total":386547056640,"available":383751147520,"internally_reserved":0,"allocated":1415626752,"data_stored":1414812032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1380173366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,3,4,12,10,7,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21889024,"num_objects":-2672,"num_object_clones":0,"num_object_copies":-10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2672,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005684"},"pg_stats":[{"pgid":"1.f","version":"38'6456","reported_seq":6569,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.790784+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:05.790784+0000","last_peered":"2024-09-10T20:24:05.790784+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:05.790784+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6494","reported_seq":6621,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:03.921228+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:03.921228+0000","last_peered":"2024-09-10T20:24:03.921228+0000","last_clean":"2024-09-10T20:24:03.921228+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:03.921228+0000","last_undegraded":"2024-09-10T20:24:03.921228+0000","last_fullsized":"2024-09-10T20:24:03.921228+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6494,"log_dups_size":0,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47169536,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6399","reported_seq":6505,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.768171+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:02.768171+0000","last_peered":"2024-09-10T20:24:02.768171+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:02.768171+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47028840,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6398,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3788}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6571","reported_seq":10735,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:03.840078+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:03.840078+0000","last_peered":"2024-09-10T20:24:03.840078+0000","last_clean":"2024-09-10T20:24:03.840078+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:03.840078+0000","last_undegraded":"2024-09-10T20:24:03.840078+0000","last_fullsized":"2024-09-10T20:24:03.840078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47800320,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":831,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6473","reported_seq":6595,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.905685+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:05.905685+0000","last_peered":"2024-09-10T20:24:05.905685+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:05.905685+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46658720,"num_objects":5696,"num_object_clones":0,"num_object_copies":22784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6407","reported_seq":6494,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.879223+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:05.879223+0000","last_peered":"2024-09-10T20:24:05.879223+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:05.879223+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46620672,"num_objects":5691,"num_object_clones":0,"num_object_copies":22764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6422","reported_seq":6507,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.250642+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:05.250642+0000","last_peered":"2024-09-10T20:24:05.250642+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:05.250642+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46265504,"num_objects":5648,"num_object_clones":0,"num_object_copies":22592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6420,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3700}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6532","reported_seq":6617,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.173956+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:05.173956+0000","last_peered":"2024-09-10T20:24:05.173956+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:05.173956+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191200,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6396","reported_seq":10363,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:05.888847+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:05.888847+0000","last_peered":"2024-09-10T20:24:05.888847+0000","last_clean":"2024-09-10T20:24:05.888847+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:05.888847+0000","last_undegraded":"2024-09-10T20:24:05.888847+0000","last_fullsized":"2024-09-10T20:24:05.888847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914704,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2949,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6406","reported_seq":8159,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.990816+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:02.990816+0000","last_peered":"2024-09-10T20:24:02.990816+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:02.990816+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47002784,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6460","reported_seq":6565,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.860509+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:05.860509+0000","last_peered":"2024-09-10T20:24:05.860509+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:05.860509+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46305008,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6479","reported_seq":6587,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.908003+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:05.908003+0000","last_peered":"2024-09-10T20:24:05.908003+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:05.908003+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6479,"log_dups_size":0,"ondisk_log_size":6479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46578256,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6478,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6252","reported_seq":6360,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.796004+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:02.796004+0000","last_peered":"2024-09-10T20:24:02.796004+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:02.796004+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6252,"log_dups_size":0,"ondisk_log_size":6252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45701712,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6438","reported_seq":6537,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.893821+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:05.893821+0000","last_peered":"2024-09-10T20:24:05.893821+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:05.893821+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331040,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3700}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6553","reported_seq":6665,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.935571+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:02.935571+0000","last_peered":"2024-09-10T20:24:02.935571+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:02.935571+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48297120,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3944}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6453","reported_seq":6542,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.109362+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:05.109362+0000","last_peered":"2024-09-10T20:24:05.109362+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:05.109362+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462112,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6451,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":747743400,"num_objects":91282,"num_object_clones":0,"num_object_copies":365128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90425,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1412644864,"data_stored":1412644864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103191,"ondisk_log_size":103191,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725496,"kb_used_data":364760,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93646344,"statfs":{"total":96636764160,"available":95893856256,"internally_reserved":0,"allocated":373514240,"data_stored":373310560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,4,2,4,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724060,"kb_used_data":368000,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93647780,"statfs":{"total":96636764160,"available":95895326720,"internally_reserved":0,"allocated":376832000,"data_stored":376628320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,4,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555172,"kb_used_data":284692,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93816668,"statfs":{"total":96636764160,"available":96068268032,"internally_reserved":0,"allocated":291524608,"data_stored":291320928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725652,"kb_used_data":364996,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93646188,"statfs":{"total":96636764160,"available":95893696512,"internally_reserved":0,"allocated":373755904,"data_stored":373552224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373334016,"data_stored":373334016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291102720,"data_stored":291102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375672832,"data_stored":375672832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":372535296,"data_stored":372535296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:08.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:09.588 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:09.588 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:09.617 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-10T20:24:07.970306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":747743400,"num_objects":91282,"num_object_clones":0,"num_object_copies":365128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90425,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103191,"ondisk_log_size":103191,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730380,"kb_used_data":1382448,"kb_used_omap":14,"kb_used_meta":1347825,"kb_avail":374756980,"statfs":{"total":386547056640,"available":383751147520,"internally_reserved":0,"allocated":1415626752,"data_stored":1414812032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1380173366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,3,4,12,10,7,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21889024,"num_objects":-2672,"num_object_clones":0,"num_object_copies":-10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2672,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005684"},"pg_stats":[{"pgid":"1.f","version":"38'6456","reported_seq":6569,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.790784+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:05.790784+0000","last_peered":"2024-09-10T20:24:05.790784+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:05.790784+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6494","reported_seq":6621,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:03.921228+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:03.921228+0000","last_peered":"2024-09-10T20:24:03.921228+0000","last_clean":"2024-09-10T20:24:03.921228+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:03.921228+0000","last_undegraded":"2024-09-10T20:24:03.921228+0000","last_fullsized":"2024-09-10T20:24:03.921228+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6494,"log_dups_size":0,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47169536,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6399","reported_seq":6505,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.768171+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:02.768171+0000","last_peered":"2024-09-10T20:24:02.768171+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:02.768171+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47028840,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6398,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3788}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6571","reported_seq":10735,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:03.840078+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:03.840078+0000","last_peered":"2024-09-10T20:24:03.840078+0000","last_clean":"2024-09-10T20:24:03.840078+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:03.840078+0000","last_undegraded":"2024-09-10T20:24:03.840078+0000","last_fullsized":"2024-09-10T20:24:03.840078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47800320,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":831,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6473","reported_seq":6595,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.905685+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:05.905685+0000","last_peered":"2024-09-10T20:24:05.905685+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:05.905685+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46658720,"num_objects":5696,"num_object_clones":0,"num_object_copies":22784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6407","reported_seq":6494,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.879223+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:05.879223+0000","last_peered":"2024-09-10T20:24:05.879223+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:05.879223+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46620672,"num_objects":5691,"num_object_clones":0,"num_object_copies":22764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6422","reported_seq":6507,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.250642+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:05.250642+0000","last_peered":"2024-09-10T20:24:05.250642+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:05.250642+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46265504,"num_objects":5648,"num_object_clones":0,"num_object_copies":22592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6420,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3700}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6532","reported_seq":6617,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.173956+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:05.173956+0000","last_peered":"2024-09-10T20:24:05.173956+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:05.173956+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191200,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6396","reported_seq":10363,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:05.888847+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:05.888847+0000","last_peered":"2024-09-10T20:24:05.888847+0000","last_clean":"2024-09-10T20:24:05.888847+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:05.888847+0000","last_undegraded":"2024-09-10T20:24:05.888847+0000","last_fullsized":"2024-09-10T20:24:05.888847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914704,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2949,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6406","reported_seq":8159,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.990816+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:02.990816+0000","last_peered":"2024-09-10T20:24:02.990816+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:02.990816+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47002784,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6460","reported_seq":6565,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.860509+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:05.860509+0000","last_peered":"2024-09-10T20:24:05.860509+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:05.860509+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46305008,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6479","reported_seq":6587,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.908003+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:05.908003+0000","last_peered":"2024-09-10T20:24:05.908003+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:05.908003+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6479,"log_dups_size":0,"ondisk_log_size":6479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46578256,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6478,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6252","reported_seq":6360,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.796004+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:02.796004+0000","last_peered":"2024-09-10T20:24:02.796004+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:02.796004+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6252,"log_dups_size":0,"ondisk_log_size":6252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45701712,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6438","reported_seq":6537,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.893821+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:05.893821+0000","last_peered":"2024-09-10T20:24:05.893821+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:05.893821+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331040,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3700}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6553","reported_seq":6665,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:02.935571+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:02.935571+0000","last_peered":"2024-09-10T20:24:02.935571+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:02.935571+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48297120,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3944}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6453","reported_seq":6542,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.109362+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:05.109362+0000","last_peered":"2024-09-10T20:24:05.109362+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:05.109362+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462112,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6451,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":747743400,"num_objects":91282,"num_object_clones":0,"num_object_copies":365128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90425,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":6807552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1412644864,"data_stored":1412644864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103191,"ondisk_log_size":103191,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725496,"kb_used_data":364760,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93646344,"statfs":{"total":96636764160,"available":95893856256,"internally_reserved":0,"allocated":373514240,"data_stored":373310560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,4,2,4,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724060,"kb_used_data":368000,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93647780,"statfs":{"total":96636764160,"available":95895326720,"internally_reserved":0,"allocated":376832000,"data_stored":376628320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,4,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555172,"kb_used_data":284692,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93816668,"statfs":{"total":96636764160,"available":96068268032,"internally_reserved":0,"allocated":291524608,"data_stored":291320928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725652,"kb_used_data":364996,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93646188,"statfs":{"total":96636764160,"available":95893696512,"internally_reserved":0,"allocated":373755904,"data_stored":373552224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373334016,"data_stored":373334016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291102720,"data_stored":291102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375672832,"data_stored":375672832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":372535296,"data_stored":372535296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:09.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:09.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6456", 'reported_seq': 6569, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.790784+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:05.790784+0000', 'last_peered': '2024-09-10T20:24:05.790784+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:05.790784+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6456, 'log_dups_size': 0, 'ondisk_log_size': 6456, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46415872, 'num_objects': 5666, 'num_object_clones': 0, 'num_object_copies': 22664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6456, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3729}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6399", 'reported_seq': 6505, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:02.768171+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:02.768171+0000', 'last_peered': '2024-09-10T20:24:02.768171+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:02.768171+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6399, 'log_dups_size': 0, 'ondisk_log_size': 6399, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47028840, 'num_objects': 5742, 'num_object_clones': 0, 'num_object_copies': 22968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5742, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6398, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3788}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6473", 'reported_seq': 6595, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.905685+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:05.905685+0000', 'last_peered': '2024-09-10T20:24:05.905685+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:05.905685+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6473, 'log_dups_size': 0, 'ondisk_log_size': 6473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46658720, 'num_objects': 5696, 'num_object_clones': 0, 'num_object_copies': 22784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6471, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6407", 'reported_seq': 6494, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.879223+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:05.879223+0000', 'last_peered': '2024-09-10T20:24:05.879223+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:05.879223+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6407, 'log_dups_size': 0, 'ondisk_log_size': 6407, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46620672, 'num_objects': 5691, 'num_object_clones': 0, 'num_object_copies': 22764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1922, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6407, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3769}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6422", 'reported_seq': 6507, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.250642+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:05.250642+0000', 'last_peered': '2024-09-10T20:24:05.250642+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:05.250642+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6422, 'log_dups_size': 0, 'ondisk_log_size': 6422, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46265504, 'num_objects': 5648, 'num_object_clones': 0, 'num_object_copies': 22592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6420, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3700}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6532", 'reported_seq': 6617, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.173956+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:05.173956+0000', 'last_peered': '2024-09-10T20:24:05.173956+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:05.173956+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6532, 'log_dups_size': 0, 'ondisk_log_size': 6532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47191200, 'num_objects': 5761, 'num_object_clones': 0, 'num_object_copies': 23044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6530, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3722}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6406", 'reported_seq': 8159, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:02.990816+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:02.990816+0000', 'last_peered': '2024-09-10T20:24:02.990816+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:02.990816+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6406, 'log_dups_size': 0, 'ondisk_log_size': 6406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47002784, 'num_objects': 5738, 'num_object_clones': 0, 'num_object_copies': 22952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6404, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 833, 'num_bytes_recovered': 6807552, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4666}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6460", 'reported_seq': 6565, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.860509+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:05.860509+0000', 'last_peered': '2024-09-10T20:24:05.860509+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:05.860509+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6460, 'log_dups_size': 0, 'ondisk_log_size': 6460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46305008, 'num_objects': 5653, 'num_object_clones': 0, 'num_object_copies': 22612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1940, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6457, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3716}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6479", 'reported_seq': 6587, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.908003+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:05.908003+0000', 'last_peered': '2024-09-10T20:24:05.908003+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:05.908003+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6479, 'log_dups_size': 0, 'ondisk_log_size': 6479, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46578256, 'num_objects': 5686, 'num_object_clones': 0, 'num_object_copies': 22744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6478, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3734}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6252", 'reported_seq': 6360, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:02.796004+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:02.796004+0000', 'last_peered': '2024-09-10T20:24:02.796004+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:02.796004+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6252, 'log_dups_size': 0, 'ondisk_log_size': 6252, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45701712, 'num_objects': 5579, 'num_object_clones': 0, 'num_object_copies': 22316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6251, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6438", 'reported_seq': 6537, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.893821+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:05.893821+0000', 'last_peered': '2024-09-10T20:24:05.893821+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:05.893821+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6438, 'log_dups_size': 0, 'ondisk_log_size': 6438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46331040, 'num_objects': 5656, 'num_object_clones': 0, 'num_object_copies': 22624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1958, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6436, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3700}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6553", 'reported_seq': 6665, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:02.935571+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:02.935571+0000', 'last_peered': '2024-09-10T20:24:02.935571+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:02.935571+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6553, 'log_dups_size': 0, 'ondisk_log_size': 6553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48297120, 'num_objects': 5896, 'num_object_clones': 0, 'num_object_copies': 23584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1954, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6551, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6453", 'reported_seq': 6542, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:05.109362+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:05.109362+0000', 'last_peered': '2024-09-10T20:24:05.109362+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:05.109362+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6453, 'log_dups_size': 0, 'ondisk_log_size': 6453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46462112, 'num_objects': 5672, 'num_object_clones': 0, 'num_object_copies': 22688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6451, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:09.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:10.357 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24549/365128 objects degraded (6.723%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":339,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91282,"data_bytes":747743400,"bytes_used":2795909120,"bytes_avail":383751147520,"bytes_total":386547056640,"degraded_objects":24549,"degraded_total":365128,"degraded_ratio":0.067233956311211415,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":104398,"recovering_keys_per_sec":0,"num_objects_recovered":234,"num_bytes_recovered":1253376,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":222},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:10.382 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:10.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:10.991 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:10.992 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:11.016 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-10T20:24:09.971462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":743622824,"num_objects":90779,"num_object_clones":0,"num_object_copies":363116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90928,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":7364608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103694,"ondisk_log_size":103694,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732360,"kb_used_data":1376120,"kb_used_omap":14,"kb_used_meta":1356145,"kb_avail":374755000,"statfs":{"total":386547056640,"available":383749120000,"internally_reserved":0,"allocated":1409146880,"data_stored":1408332160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1388693046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,3,4,22,10,10,1,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26009600,"num_objects":-3175,"num_object_clones":0,"num_object_copies":-12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3175,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":308,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004726"},"pg_stats":[{"pgid":"1.f","version":"38'6456","reported_seq":6569,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.790784+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:05.790784+0000","last_peered":"2024-09-10T20:24:05.790784+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:05.790784+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6579","reported_seq":6706,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:08.689448+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:08.689448+0000","last_peered":"2024-09-10T20:24:08.689448+0000","last_clean":"2024-09-10T20:24:08.689448+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:08.689448+0000","last_undegraded":"2024-09-10T20:24:08.689448+0000","last_fullsized":"2024-09-10T20:24:08.689448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46473216,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6491","reported_seq":6601,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:08.030139+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:08.030139+0000","last_peered":"2024-09-10T20:24:08.030139+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:08.030139+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46275176,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6490,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3698}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6639","reported_seq":10803,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:08.569643+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:08.569643+0000","last_peered":"2024-09-10T20:24:08.569643+0000","last_clean":"2024-09-10T20:24:08.569643+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:08.569643+0000","last_undegraded":"2024-09-10T20:24:08.569643+0000","last_fullsized":"2024-09-10T20:24:08.569643+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47243264,"num_objects":5767,"num_object_clones":0,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6473","reported_seq":6595,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.905685+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:05.905685+0000","last_peered":"2024-09-10T20:24:05.905685+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:05.905685+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46658720,"num_objects":5696,"num_object_clones":0,"num_object_copies":22784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6407","reported_seq":6494,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.879223+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:05.879223+0000","last_peered":"2024-09-10T20:24:05.879223+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:05.879223+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46620672,"num_objects":5691,"num_object_clones":0,"num_object_copies":22764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6422","reported_seq":6507,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.250642+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:05.250642+0000","last_peered":"2024-09-10T20:24:05.250642+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:05.250642+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46265504,"num_objects":5648,"num_object_clones":0,"num_object_copies":22592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6420,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3700}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6532","reported_seq":6617,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.173956+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:05.173956+0000","last_peered":"2024-09-10T20:24:05.173956+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:05.173956+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191200,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6396","reported_seq":10363,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:05.888847+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:05.888847+0000","last_peered":"2024-09-10T20:24:05.888847+0000","last_clean":"2024-09-10T20:24:05.888847+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:05.888847+0000","last_undegraded":"2024-09-10T20:24:05.888847+0000","last_fullsized":"2024-09-10T20:24:05.888847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914704,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2949,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6488","reported_seq":8377,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:07.987271+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:07.987271+0000","last_peered":"2024-09-10T20:24:07.987271+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:07.987271+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6488,"log_dups_size":0,"ondisk_log_size":6488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331040,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6486,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":901,"num_bytes_recovered":7364608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4652}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6460","reported_seq":6565,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.860509+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:05.860509+0000","last_peered":"2024-09-10T20:24:05.860509+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:05.860509+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46305008,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6479","reported_seq":6587,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.908003+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:05.908003+0000","last_peered":"2024-09-10T20:24:05.908003+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:05.908003+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6479,"log_dups_size":0,"ondisk_log_size":6479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46578256,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6478,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6344","reported_seq":6456,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:08.036951+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:08.036951+0000","last_peered":"2024-09-10T20:24:08.036951+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:08.036951+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44948048,"num_objects":5487,"num_object_clones":0,"num_object_copies":21948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6343,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6438","reported_seq":6537,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.893821+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:05.893821+0000","last_peered":"2024-09-10T20:24:05.893821+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:05.893821+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331040,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3700}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6637","reported_seq":6753,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:07.962142+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:07.962142+0000","last_peered":"2024-09-10T20:24:07.962142+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:07.962142+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6637,"log_dups_size":0,"ondisk_log_size":6637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47608992,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6635,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3862}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6453","reported_seq":6542,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:05.109362+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:05.109362+0000","last_peered":"2024-09-10T20:24:05.109362+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:05.109362+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462112,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6451,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":743622824,"num_objects":90779,"num_object_clones":0,"num_object_copies":363116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90928,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":7364608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1406074880,"data_stored":1406074880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103694,"ondisk_log_size":103694,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725496,"kb_used_data":364760,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93646344,"statfs":{"total":96636764160,"available":95893856256,"internally_reserved":0,"allocated":373514240,"data_stored":373310560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,4,2,4,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722944,"kb_used_data":362736,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93648896,"statfs":{"total":96636764160,"available":95896469504,"internally_reserved":0,"allocated":371441664,"data_stored":371237984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558268,"kb_used_data":283628,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93813572,"statfs":{"total":96636764160,"available":96065097728,"internally_reserved":0,"allocated":290435072,"data_stored":290231392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725652,"kb_used_data":364996,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93646188,"statfs":{"total":96636764160,"available":95893696512,"internally_reserved":0,"allocated":373755904,"data_stored":373552224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373334016,"data_stored":373334016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290058240,"data_stored":290058240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370147328,"data_stored":370147328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":372535296,"data_stored":372535296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:14.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:14.527 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:14.527 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:14.553 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-10T20:24:13.972832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733980840,"num_objects":89602,"num_object_clones":0,"num_object_copies":358408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92105,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104871,"ondisk_log_size":104871,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2728956,"kb_used_data":1360220,"kb_used_omap":14,"kb_used_meta":1368625,"kb_avail":374758404,"statfs":{"total":386547056640,"available":383752605696,"internally_reserved":0,"allocated":1392865280,"data_stored":1392050560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1401472566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,10,7,1,2,0,0,1,3,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24829952,"num_objects":-3031,"num_object_clones":0,"num_object_copies":-12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":1916928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004617"},"pg_stats":[{"pgid":"1.f","version":"38'6550","reported_seq":6671,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.980444+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:10.980444+0000","last_peered":"2024-09-10T20:24:10.980444+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:10.980444+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45645824,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6579","reported_seq":6706,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:08.689448+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:08.689448+0000","last_peered":"2024-09-10T20:24:08.689448+0000","last_clean":"2024-09-10T20:24:08.689448+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:08.689448+0000","last_undegraded":"2024-09-10T20:24:08.689448+0000","last_fullsized":"2024-09-10T20:24:08.689448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46473216,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6563","reported_seq":6676,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:12.807617+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:12.807617+0000","last_peered":"2024-09-10T20:24:12.807617+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:12.807617+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45685352,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6562,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3629}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6639","reported_seq":10803,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:08.569643+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:08.569643+0000","last_peered":"2024-09-10T20:24:08.569643+0000","last_clean":"2024-09-10T20:24:08.569643+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:08.569643+0000","last_undegraded":"2024-09-10T20:24:08.569643+0000","last_fullsized":"2024-09-10T20:24:08.569643+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47243264,"num_objects":5767,"num_object_clones":0,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6565","reported_seq":6692,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.866585+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:10.866585+0000","last_peered":"2024-09-10T20:24:10.866585+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:10.866585+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6563,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6491","reported_seq":6586,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.973102+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:10.973102+0000","last_peered":"2024-09-10T20:24:10.973102+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:10.973102+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45932544,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6530","reported_seq":6626,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.278859+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:10.278859+0000","last_peered":"2024-09-10T20:24:10.278859+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:10.278859+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45380768,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3596}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6624","reported_seq":6713,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.102425+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:10.102425+0000","last_peered":"2024-09-10T20:24:10.102425+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:10.102425+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46437536,"num_objects":5669,"num_object_clones":0,"num_object_copies":22676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6487","reported_seq":10453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:10.957011+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:10.957011+0000","last_peered":"2024-09-10T20:24:10.957011+0000","last_clean":"2024-09-10T20:24:10.957011+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:10.957011+0000","last_undegraded":"2024-09-10T20:24:10.957011+0000","last_fullsized":"2024-09-10T20:24:10.957011+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45169232,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6565","reported_seq":8664,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.026517+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:13.026517+0000","last_peered":"2024-09-10T20:24:13.026517+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:13.026517+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700256,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6563,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4680}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6549","reported_seq":6657,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.992238+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:10.992238+0000","last_peered":"2024-09-10T20:24:10.992238+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:10.992238+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6549,"log_dups_size":0,"ondisk_log_size":6549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45575920,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6546,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6568","reported_seq":6676,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.908940+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:10.908940+0000","last_peered":"2024-09-10T20:24:10.908940+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:10.908940+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6568,"log_dups_size":0,"ondisk_log_size":6568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45849168,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6567,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6419","reported_seq":6533,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.018196+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:13.018196+0000","last_peered":"2024-09-10T20:24:13.018196+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:13.018196+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44333648,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6418,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6513","reported_seq":6615,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.740137+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:10.740137+0000","last_peered":"2024-09-10T20:24:10.740137+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:10.740137+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716640,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3625}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6708","reported_seq":6832,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.056154+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:13.056154+0000","last_peered":"2024-09-10T20:24:13.056154+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:13.056154+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6708,"log_dups_size":0,"ondisk_log_size":6708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47027360,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6521","reported_seq":6610,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.249897+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:10.249897+0000","last_peered":"2024-09-10T20:24:10.249897+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:10.249897+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733980840,"num_objects":89602,"num_object_clones":0,"num_object_copies":358408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92105,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1389944832,"data_stored":1389944832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104871,"ondisk_log_size":104871,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724268,"kb_used_data":359372,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93647572,"statfs":{"total":96636764160,"available":95895113728,"internally_reserved":0,"allocated":367996928,"data_stored":367793248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,2,0,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722100,"kb_used_data":357716,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93649740,"statfs":{"total":96636764160,"available":95897333760,"internally_reserved":0,"allocated":366301184,"data_stored":366097504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558268,"kb_used_data":283628,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93813572,"statfs":{"total":96636764160,"available":96065097728,"internally_reserved":0,"allocated":290435072,"data_stored":290231392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724320,"kb_used_data":359504,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93647520,"statfs":{"total":96636764160,"available":95895060480,"internally_reserved":0,"allocated":368132096,"data_stored":367928416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,1,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367755264,"data_stored":367755264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290058240,"data_stored":290058240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365051904,"data_stored":365051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367079424,"data_stored":367079424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:14.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:15.130 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:15.131 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:15.149 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-10T20:24:13.972832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733980840,"num_objects":89602,"num_object_clones":0,"num_object_copies":358408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92105,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104871,"ondisk_log_size":104871,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2728956,"kb_used_data":1360220,"kb_used_omap":14,"kb_used_meta":1368625,"kb_avail":374758404,"statfs":{"total":386547056640,"available":383752605696,"internally_reserved":0,"allocated":1392865280,"data_stored":1392050560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1401472566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,10,7,1,2,0,0,1,3,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24829952,"num_objects":-3031,"num_object_clones":0,"num_object_copies":-12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":1916928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004617"},"pg_stats":[{"pgid":"1.f","version":"38'6550","reported_seq":6671,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.980444+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:10.980444+0000","last_peered":"2024-09-10T20:24:10.980444+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:10.980444+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45645824,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6579","reported_seq":6706,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:08.689448+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:08.689448+0000","last_peered":"2024-09-10T20:24:08.689448+0000","last_clean":"2024-09-10T20:24:08.689448+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:08.689448+0000","last_undegraded":"2024-09-10T20:24:08.689448+0000","last_fullsized":"2024-09-10T20:24:08.689448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46473216,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6563","reported_seq":6676,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:12.807617+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:12.807617+0000","last_peered":"2024-09-10T20:24:12.807617+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:12.807617+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45685352,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6562,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3629}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6639","reported_seq":10803,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:08.569643+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:08.569643+0000","last_peered":"2024-09-10T20:24:08.569643+0000","last_clean":"2024-09-10T20:24:08.569643+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:08.569643+0000","last_undegraded":"2024-09-10T20:24:08.569643+0000","last_fullsized":"2024-09-10T20:24:08.569643+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47243264,"num_objects":5767,"num_object_clones":0,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6565","reported_seq":6692,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.866585+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:10.866585+0000","last_peered":"2024-09-10T20:24:10.866585+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:10.866585+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6563,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6491","reported_seq":6586,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.973102+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:10.973102+0000","last_peered":"2024-09-10T20:24:10.973102+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:10.973102+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45932544,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6530","reported_seq":6626,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.278859+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:10.278859+0000","last_peered":"2024-09-10T20:24:10.278859+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:10.278859+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45380768,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3596}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6624","reported_seq":6713,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.102425+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:10.102425+0000","last_peered":"2024-09-10T20:24:10.102425+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:10.102425+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46437536,"num_objects":5669,"num_object_clones":0,"num_object_copies":22676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6487","reported_seq":10453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:10.957011+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:10.957011+0000","last_peered":"2024-09-10T20:24:10.957011+0000","last_clean":"2024-09-10T20:24:10.957011+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:10.957011+0000","last_undegraded":"2024-09-10T20:24:10.957011+0000","last_fullsized":"2024-09-10T20:24:10.957011+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45169232,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6565","reported_seq":8664,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.026517+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:13.026517+0000","last_peered":"2024-09-10T20:24:13.026517+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:13.026517+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700256,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6563,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4680}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6549","reported_seq":6657,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.992238+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:10.992238+0000","last_peered":"2024-09-10T20:24:10.992238+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:10.992238+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6549,"log_dups_size":0,"ondisk_log_size":6549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45575920,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6546,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6568","reported_seq":6676,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.908940+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:10.908940+0000","last_peered":"2024-09-10T20:24:10.908940+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:10.908940+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6568,"log_dups_size":0,"ondisk_log_size":6568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45849168,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6567,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6419","reported_seq":6533,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.018196+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:13.018196+0000","last_peered":"2024-09-10T20:24:13.018196+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:13.018196+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44333648,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6418,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6513","reported_seq":6615,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.740137+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:10.740137+0000","last_peered":"2024-09-10T20:24:10.740137+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:10.740137+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716640,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3625}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6708","reported_seq":6832,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.056154+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:13.056154+0000","last_peered":"2024-09-10T20:24:13.056154+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:13.056154+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6708,"log_dups_size":0,"ondisk_log_size":6708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47027360,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6521","reported_seq":6610,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.249897+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:10.249897+0000","last_peered":"2024-09-10T20:24:10.249897+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:10.249897+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733980840,"num_objects":89602,"num_object_clones":0,"num_object_copies":358408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92105,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1389944832,"data_stored":1389944832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104871,"ondisk_log_size":104871,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724268,"kb_used_data":359372,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93647572,"statfs":{"total":96636764160,"available":95895113728,"internally_reserved":0,"allocated":367996928,"data_stored":367793248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,2,0,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722100,"kb_used_data":357716,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93649740,"statfs":{"total":96636764160,"available":95897333760,"internally_reserved":0,"allocated":366301184,"data_stored":366097504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558268,"kb_used_data":283628,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93813572,"statfs":{"total":96636764160,"available":96065097728,"internally_reserved":0,"allocated":290435072,"data_stored":290231392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724320,"kb_used_data":359504,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93647520,"statfs":{"total":96636764160,"available":95895060480,"internally_reserved":0,"allocated":368132096,"data_stored":367928416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,1,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367755264,"data_stored":367755264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290058240,"data_stored":290058240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365051904,"data_stored":365051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367079424,"data_stored":367079424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6550", 'reported_seq': 6671, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.980444+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:10.980444+0000', 'last_peered': '2024-09-10T20:24:10.980444+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:10.980444+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6550, 'log_dups_size': 0, 'ondisk_log_size': 6550, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45645824, 'num_objects': 5572, 'num_object_clones': 0, 'num_object_copies': 22288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1933, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6550, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3639}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6563", 'reported_seq': 6676, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:12.807617+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:12.807617+0000', 'last_peered': '2024-09-10T20:24:12.807617+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:12.807617+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6563, 'log_dups_size': 0, 'ondisk_log_size': 6563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45685352, 'num_objects': 5578, 'num_object_clones': 0, 'num_object_copies': 22312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5578, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6562, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3629}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6565", 'reported_seq': 6692, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.866585+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:10.866585+0000', 'last_peered': '2024-09-10T20:24:10.866585+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:10.866585+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6565, 'log_dups_size': 0, 'ondisk_log_size': 6565, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45905056, 'num_objects': 5604, 'num_object_clones': 0, 'num_object_copies': 22416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1951, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6563, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3655}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:15.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6491", 'reported_seq': 6586, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.973102+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:10.973102+0000', 'last_peered': '2024-09-10T20:24:10.973102+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:10.973102+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6491, 'log_dups_size': 0, 'ondisk_log_size': 6491, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45932544, 'num_objects': 5607, 'num_object_clones': 0, 'num_object_copies': 22428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1918, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6491, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3689}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6530", 'reported_seq': 6626, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.278859+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:10.278859+0000', 'last_peered': '2024-09-10T20:24:10.278859+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:10.278859+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6530, 'log_dups_size': 0, 'ondisk_log_size': 6530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45380768, 'num_objects': 5540, 'num_object_clones': 0, 'num_object_copies': 22160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1946, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6528, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3596}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6624", 'reported_seq': 6713, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.102425+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:10.102425+0000', 'last_peered': '2024-09-10T20:24:10.102425+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:10.102425+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6624, 'log_dups_size': 0, 'ondisk_log_size': 6624, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46437536, 'num_objects': 5669, 'num_object_clones': 0, 'num_object_copies': 22676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6622, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6565", 'reported_seq': 8664, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:13.026517+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:13.026517+0000', 'last_peered': '2024-09-10T20:24:13.026517+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:13.026517+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6565, 'log_dups_size': 0, 'ondisk_log_size': 6565, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45700256, 'num_objects': 5579, 'num_object_clones': 0, 'num_object_copies': 22316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6563, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1006, 'num_bytes_recovered': 8224768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4680}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6549", 'reported_seq': 6657, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.992238+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:10.992238+0000', 'last_peered': '2024-09-10T20:24:10.992238+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:10.992238+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6549, 'log_dups_size': 0, 'ondisk_log_size': 6549, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45575920, 'num_objects': 5564, 'num_object_clones': 0, 'num_object_copies': 22256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6546, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3628}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:15.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6568", 'reported_seq': 6676, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.908940+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:10.908940+0000', 'last_peered': '2024-09-10T20:24:10.908940+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:10.908940+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6568, 'log_dups_size': 0, 'ondisk_log_size': 6568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45849168, 'num_objects': 5597, 'num_object_clones': 0, 'num_object_copies': 22388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6567, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3645}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6419", 'reported_seq': 6533, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:13.018196+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:13.018196+0000', 'last_peered': '2024-09-10T20:24:13.018196+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:13.018196+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6419, 'log_dups_size': 0, 'ondisk_log_size': 6419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44333648, 'num_objects': 5412, 'num_object_clones': 0, 'num_object_copies': 21648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1936, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6418, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6513", 'reported_seq': 6615, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.740137+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:10.740137+0000', 'last_peered': '2024-09-10T20:24:10.740137+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:10.740137+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6513, 'log_dups_size': 0, 'ondisk_log_size': 6513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45716640, 'num_objects': 5581, 'num_object_clones': 0, 'num_object_copies': 22324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1958, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6511, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3625}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6708", 'reported_seq': 6832, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:13.056154+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:13.056154+0000', 'last_peered': '2024-09-10T20:24:13.056154+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:13.056154+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6708, 'log_dups_size': 0, 'ondisk_log_size': 6708, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47027360, 'num_objects': 5741, 'num_object_clones': 0, 'num_object_copies': 22964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1948, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6706, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:15.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6521", 'reported_seq': 6610, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:10.249897+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:10.249897+0000', 'last_peered': '2024-09-10T20:24:10.249897+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:10.249897+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6521, 'log_dups_size': 0, 'ondisk_log_size': 6521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45905056, 'num_objects': 5604, 'num_object_clones': 0, 'num_object_copies': 22416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6519, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:15.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:15.866 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:15.866 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24345/358408 objects degraded (6.793%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":344,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89602,"data_bytes":733980840,"bytes_used":2794450944,"bytes_avail":383752605696,"bytes_total":386547056640,"degraded_objects":24345,"degraded_total":358408,"degraded_ratio":0.067925381129885498,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":159682,"recovering_keys_per_sec":0,"num_objects_recovered":313,"num_bytes_recovered":1916928,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":252},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:15.887 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:16.502 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:16.529 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-10T20:24:15.973585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":730966184,"num_objects":89234,"num_object_clones":0,"num_object_copies":356936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92473,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105239,"ondisk_log_size":105239,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698516,"kb_used_data":1354736,"kb_used_omap":14,"kb_used_meta":1343665,"kb_avail":374788844,"statfs":{"total":386547056640,"available":383783776256,"internally_reserved":0,"allocated":1387249664,"data_stored":1386434944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1375913526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,6,9,2,1,0,0,1,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25075712,"num_objects":-3061,"num_object_clones":0,"num_object_copies":-12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004852"},"pg_stats":[{"pgid":"1.f","version":"38'6550","reported_seq":6671,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.980444+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:10.980444+0000","last_peered":"2024-09-10T20:24:10.980444+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:10.980444+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45645824,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6635","reported_seq":6762,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:13.752452+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:13.752452+0000","last_peered":"2024-09-10T20:24:13.752452+0000","last_clean":"2024-09-10T20:24:13.752452+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:13.752452+0000","last_undegraded":"2024-09-10T20:24:13.752452+0000","last_fullsized":"2024-09-10T20:24:13.752452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46014464,"num_objects":5617,"num_object_clones":0,"num_object_copies":22468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3076,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6563","reported_seq":6676,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:12.807617+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:12.807617+0000","last_peered":"2024-09-10T20:24:12.807617+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:12.807617+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45685352,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6562,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3629}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6725","reported_seq":10889,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:13.809300+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:13.809300+0000","last_peered":"2024-09-10T20:24:13.809300+0000","last_clean":"2024-09-10T20:24:13.809300+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:13.809300+0000","last_undegraded":"2024-09-10T20:24:13.809300+0000","last_fullsized":"2024-09-10T20:24:13.809300+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6725,"log_dups_size":0,"ondisk_log_size":6725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46538752,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":985,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6565","reported_seq":6692,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.866585+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:10.866585+0000","last_peered":"2024-09-10T20:24:10.866585+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:10.866585+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6563,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6491","reported_seq":6586,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.973102+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:10.973102+0000","last_peered":"2024-09-10T20:24:10.973102+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:10.973102+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45932544,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6602","reported_seq":6706,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.290074+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:15.290074+0000","last_peered":"2024-09-10T20:24:15.290074+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:15.290074+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44790944,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6600,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3528}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6711","reported_seq":6810,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.216716+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:15.216716+0000","last_peered":"2024-09-10T20:24:15.216716+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:15.216716+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45724832,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6709,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6487","reported_seq":10453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:10.957011+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:10.957011+0000","last_peered":"2024-09-10T20:24:10.957011+0000","last_clean":"2024-09-10T20:24:10.957011+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:10.957011+0000","last_undegraded":"2024-09-10T20:24:10.957011+0000","last_fullsized":"2024-09-10T20:24:10.957011+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45169232,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6565","reported_seq":8664,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.026517+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:13.026517+0000","last_peered":"2024-09-10T20:24:13.026517+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:13.026517+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700256,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6563,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4680}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6549","reported_seq":6657,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.992238+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:10.992238+0000","last_peered":"2024-09-10T20:24:10.992238+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:10.992238+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6549,"log_dups_size":0,"ondisk_log_size":6549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45575920,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6546,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6568","reported_seq":6676,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.908940+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:10.908940+0000","last_peered":"2024-09-10T20:24:10.908940+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:10.908940+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6568,"log_dups_size":0,"ondisk_log_size":6568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45849168,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6567,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6419","reported_seq":6533,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.018196+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:13.018196+0000","last_peered":"2024-09-10T20:24:13.018196+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:13.018196+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44333648,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6418,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6513","reported_seq":6615,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:10.740137+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:10.740137+0000","last_peered":"2024-09-10T20:24:10.740137+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:10.740137+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716640,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3625}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6708","reported_seq":6832,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:13.056154+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:13.056154+0000","last_peered":"2024-09-10T20:24:13.056154+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:13.056154+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6708,"log_dups_size":0,"ondisk_log_size":6708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47027360,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6588","reported_seq":6683,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.267754+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:15.267754+0000","last_peered":"2024-09-10T20:24:15.267754+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:15.267754+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45356192,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":730966184,"num_objects":89234,"num_object_clones":0,"num_object_copies":356936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92473,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384275968,"data_stored":1384275968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105239,"ondisk_log_size":105239,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724268,"kb_used_data":359372,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93647572,"statfs":{"total":96636764160,"available":95895113728,"internally_reserved":0,"allocated":367996928,"data_stored":367793248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,2,0,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722100,"kb_used_data":357716,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93649740,"statfs":{"total":96636764160,"available":95897333760,"internally_reserved":0,"allocated":366301184,"data_stored":366097504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545136,"kb_used_data":282976,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93826704,"statfs":{"total":96636764160,"available":96078544896,"internally_reserved":0,"allocated":289767424,"data_stored":289563744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":268431058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707012,"kb_used_data":354672,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93664828,"statfs":{"total":96636764160,"available":95912783872,"internally_reserved":0,"allocated":363184128,"data_stored":362980448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,0,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362819584,"data_stored":362819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289325056,"data_stored":289325056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365051904,"data_stored":365051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367079424,"data_stored":367079424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:19.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:20.171 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:20.172 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:20.192 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-10T20:24:19.975742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":722938024,"num_objects":88254,"num_object_clones":0,"num_object_copies":353016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93453,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106219,"ondisk_log_size":106219,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707640,"kb_used_data":1344416,"kb_used_omap":14,"kb_used_meta":1363121,"kb_avail":374779720,"statfs":{"total":386547056640,"available":383774433280,"internally_reserved":0,"allocated":1376681984,"data_stored":1375867264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1395836470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,9,6,9,5,1,0,11,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24805376,"num_objects":-3028,"num_object_clones":0,"num_object_copies":-12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005435"},"pg_stats":[{"pgid":"1.f","version":"38'6629","reported_seq":6750,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:16.003448+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:16.003448+0000","last_peered":"2024-09-10T20:24:16.003448+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:16.003448+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6629,"log_dups_size":0,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44998656,"num_objects":5493,"num_object_clones":0,"num_object_copies":21972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6713","reported_seq":6840,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.891948+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:18.891948+0000","last_peered":"2024-09-10T20:24:18.891948+0000","last_clean":"2024-09-10T20:24:18.891948+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:18.891948+0000","last_undegraded":"2024-09-10T20:24:18.891948+0000","last_fullsized":"2024-09-10T20:24:18.891948+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45375488,"num_objects":5539,"num_object_clones":0,"num_object_copies":22156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3154,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6652","reported_seq":6778,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:17.912318+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:17.912318+0000","last_peered":"2024-09-10T20:24:17.912318+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:17.912318+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6652,"log_dups_size":0,"ondisk_log_size":6652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44956264,"num_objects":5489,"num_object_clones":0,"num_object_copies":21956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6651,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6803","reported_seq":10967,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.869548+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:18.869548+0000","last_peered":"2024-09-10T20:24:18.869548+0000","last_clean":"2024-09-10T20:24:18.869548+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:18.869548+0000","last_undegraded":"2024-09-10T20:24:18.869548+0000","last_fullsized":"2024-09-10T20:24:18.869548+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6803,"log_dups_size":0,"ondisk_log_size":6803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45899776,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1063,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6645","reported_seq":6777,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.958918+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:15.958918+0000","last_peered":"2024-09-10T20:24:15.958918+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:15.958918+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249696,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6558","reported_seq":6655,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:16.003269+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:16.003269+0000","last_peered":"2024-09-10T20:24:16.003269+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:16.003269+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45383680,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3623}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6602","reported_seq":6706,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.290074+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:15.290074+0000","last_peered":"2024-09-10T20:24:15.290074+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:15.290074+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44790944,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6600,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3528}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6711","reported_seq":6810,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.216716+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:15.216716+0000","last_peered":"2024-09-10T20:24:15.216716+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:15.216716+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45724832,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6709,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6553","reported_seq":10520,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:15.845165+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:15.845165+0000","last_peered":"2024-09-10T20:24:15.845165+0000","last_clean":"2024-09-10T20:24:15.845165+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:15.845165+0000","last_undegraded":"2024-09-10T20:24:15.845165+0000","last_fullsized":"2024-09-10T20:24:15.845165+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44628560,"num_objects":5448,"num_object_clones":0,"num_object_copies":21792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3106,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6625","reported_seq":8930,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:18.079689+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:18.079689+0000","last_peered":"2024-09-10T20:24:18.079689+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:18.079689+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208736,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6623,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6611","reported_seq":6723,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.978083+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:15.978083+0000","last_peered":"2024-09-10T20:24:15.978083+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:15.978083+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45068016,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6656","reported_seq":6779,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.927400+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:15.927400+0000","last_peered":"2024-09-10T20:24:15.927400+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:15.927400+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6656,"log_dups_size":0,"ondisk_log_size":6656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45128272,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6655,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3564}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6480","reported_seq":6598,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:17.965234+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:17.965234+0000","last_peered":"2024-09-10T20:24:17.965234+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:17.965234+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43833936,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6479,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6600","reported_seq":6705,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.877100+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:15.877100+0000","last_peered":"2024-09-10T20:24:15.877100+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:15.877100+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6600,"log_dups_size":0,"ondisk_log_size":6600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45003936,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6598,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3540}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6793","reported_seq":6919,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:18.018079+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:18.018079+0000","last_peered":"2024-09-10T20:24:18.018079+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:18.018079+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331040,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6588","reported_seq":6683,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.267754+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:15.267754+0000","last_peered":"2024-09-10T20:24:15.267754+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:15.267754+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45356192,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":722938024,"num_objects":88254,"num_object_clones":0,"num_object_copies":353016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93453,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1373581312,"data_stored":1373581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106219,"ondisk_log_size":106219,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707004,"kb_used_data":354600,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93664836,"statfs":{"total":96636764160,"available":95912792064,"internally_reserved":0,"allocated":363110400,"data_stored":362906720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,2,3,1,1,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727084,"kb_used_data":352976,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93644756,"statfs":{"total":96636764160,"available":95892230144,"internally_reserved":0,"allocated":361447424,"data_stored":361243744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,4,3,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566540,"kb_used_data":282168,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93805300,"statfs":{"total":96636764160,"available":96056627200,"internally_reserved":0,"allocated":288940032,"data_stored":288736352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707012,"kb_used_data":354672,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93664828,"statfs":{"total":96636764160,"available":95912783872,"internally_reserved":0,"allocated":363184128,"data_stored":362980448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,0,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362819584,"data_stored":362819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288546816,"data_stored":288546816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360034304,"data_stored":360034304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362180608,"data_stored":362180608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:20.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:20.789 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:20.789 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:20.818 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-10T20:24:19.975742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":722938024,"num_objects":88254,"num_object_clones":0,"num_object_copies":353016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93453,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106219,"ondisk_log_size":106219,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707640,"kb_used_data":1344416,"kb_used_omap":14,"kb_used_meta":1363121,"kb_avail":374779720,"statfs":{"total":386547056640,"available":383774433280,"internally_reserved":0,"allocated":1376681984,"data_stored":1375867264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1395836470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,9,6,9,5,1,0,11,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24805376,"num_objects":-3028,"num_object_clones":0,"num_object_copies":-12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005435"},"pg_stats":[{"pgid":"1.f","version":"38'6629","reported_seq":6750,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:16.003448+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:16.003448+0000","last_peered":"2024-09-10T20:24:16.003448+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:16.003448+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6629,"log_dups_size":0,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44998656,"num_objects":5493,"num_object_clones":0,"num_object_copies":21972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6713","reported_seq":6840,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.891948+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:18.891948+0000","last_peered":"2024-09-10T20:24:18.891948+0000","last_clean":"2024-09-10T20:24:18.891948+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:18.891948+0000","last_undegraded":"2024-09-10T20:24:18.891948+0000","last_fullsized":"2024-09-10T20:24:18.891948+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45375488,"num_objects":5539,"num_object_clones":0,"num_object_copies":22156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3154,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6652","reported_seq":6778,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:17.912318+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:17.912318+0000","last_peered":"2024-09-10T20:24:17.912318+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:17.912318+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6652,"log_dups_size":0,"ondisk_log_size":6652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44956264,"num_objects":5489,"num_object_clones":0,"num_object_copies":21956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6651,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6803","reported_seq":10967,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.869548+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:18.869548+0000","last_peered":"2024-09-10T20:24:18.869548+0000","last_clean":"2024-09-10T20:24:18.869548+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:18.869548+0000","last_undegraded":"2024-09-10T20:24:18.869548+0000","last_fullsized":"2024-09-10T20:24:18.869548+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6803,"log_dups_size":0,"ondisk_log_size":6803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45899776,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1063,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6645","reported_seq":6777,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.958918+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:15.958918+0000","last_peered":"2024-09-10T20:24:15.958918+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:15.958918+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249696,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6558","reported_seq":6655,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:16.003269+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:16.003269+0000","last_peered":"2024-09-10T20:24:16.003269+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:16.003269+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45383680,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3623}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6602","reported_seq":6706,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.290074+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:15.290074+0000","last_peered":"2024-09-10T20:24:15.290074+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:15.290074+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44790944,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6600,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3528}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6711","reported_seq":6810,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.216716+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:15.216716+0000","last_peered":"2024-09-10T20:24:15.216716+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:15.216716+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45724832,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6709,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6553","reported_seq":10520,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:15.845165+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:15.845165+0000","last_peered":"2024-09-10T20:24:15.845165+0000","last_clean":"2024-09-10T20:24:15.845165+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:15.845165+0000","last_undegraded":"2024-09-10T20:24:15.845165+0000","last_fullsized":"2024-09-10T20:24:15.845165+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44628560,"num_objects":5448,"num_object_clones":0,"num_object_copies":21792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3106,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6625","reported_seq":8930,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:18.079689+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:18.079689+0000","last_peered":"2024-09-10T20:24:18.079689+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:18.079689+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208736,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6623,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6611","reported_seq":6723,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.978083+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:15.978083+0000","last_peered":"2024-09-10T20:24:15.978083+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:15.978083+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45068016,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6656","reported_seq":6779,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.927400+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:15.927400+0000","last_peered":"2024-09-10T20:24:15.927400+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:15.927400+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6656,"log_dups_size":0,"ondisk_log_size":6656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45128272,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6655,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3564}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6480","reported_seq":6598,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:17.965234+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:17.965234+0000","last_peered":"2024-09-10T20:24:17.965234+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:17.965234+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43833936,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6479,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6600","reported_seq":6705,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.877100+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:15.877100+0000","last_peered":"2024-09-10T20:24:15.877100+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:15.877100+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6600,"log_dups_size":0,"ondisk_log_size":6600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45003936,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6598,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3540}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6793","reported_seq":6919,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:18.018079+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:18.018079+0000","last_peered":"2024-09-10T20:24:18.018079+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:18.018079+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331040,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6588","reported_seq":6683,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:15.267754+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:15.267754+0000","last_peered":"2024-09-10T20:24:15.267754+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:15.267754+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45356192,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":722938024,"num_objects":88254,"num_object_clones":0,"num_object_copies":353016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93453,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1373581312,"data_stored":1373581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106219,"ondisk_log_size":106219,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707004,"kb_used_data":354600,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93664836,"statfs":{"total":96636764160,"available":95912792064,"internally_reserved":0,"allocated":363110400,"data_stored":362906720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,2,3,1,1,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727084,"kb_used_data":352976,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93644756,"statfs":{"total":96636764160,"available":95892230144,"internally_reserved":0,"allocated":361447424,"data_stored":361243744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,4,3,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566540,"kb_used_data":282168,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93805300,"statfs":{"total":96636764160,"available":96056627200,"internally_reserved":0,"allocated":288940032,"data_stored":288736352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707012,"kb_used_data":354672,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93664828,"statfs":{"total":96636764160,"available":95912783872,"internally_reserved":0,"allocated":363184128,"data_stored":362980448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,0,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362819584,"data_stored":362819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288546816,"data_stored":288546816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360034304,"data_stored":360034304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362180608,"data_stored":362180608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:20.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:20.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6629", 'reported_seq': 6750, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:16.003448+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:16.003448+0000', 'last_peered': '2024-09-10T20:24:16.003448+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:16.003448+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6629, 'log_dups_size': 0, 'ondisk_log_size': 6629, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44998656, 'num_objects': 5493, 'num_object_clones': 0, 'num_object_copies': 21972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1933, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6629, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3560}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:20.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:20.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6652", 'reported_seq': 6778, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:17.912318+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:17.912318+0000', 'last_peered': '2024-09-10T20:24:17.912318+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:17.912318+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6652, 'log_dups_size': 0, 'ondisk_log_size': 6652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44956264, 'num_objects': 5489, 'num_object_clones': 0, 'num_object_copies': 21956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5489, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6651, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3545}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:20.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:20.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6645", 'reported_seq': 6777, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:15.958918+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:15.958918+0000', 'last_peered': '2024-09-10T20:24:15.958918+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:15.958918+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6645, 'log_dups_size': 0, 'ondisk_log_size': 6645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45249696, 'num_objects': 5524, 'num_object_clones': 0, 'num_object_copies': 22096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1948, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6643, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3578}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6558", 'reported_seq': 6655, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:16.003269+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:16.003269+0000', 'last_peered': '2024-09-10T20:24:16.003269+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:16.003269+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6558, 'log_dups_size': 0, 'ondisk_log_size': 6558, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45383680, 'num_objects': 5540, 'num_object_clones': 0, 'num_object_copies': 22160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1917, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6558, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3623}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6602", 'reported_seq': 6706, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:15.290074+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:15.290074+0000', 'last_peered': '2024-09-10T20:24:15.290074+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:15.290074+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6602, 'log_dups_size': 0, 'ondisk_log_size': 6602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44790944, 'num_objects': 5468, 'num_object_clones': 0, 'num_object_copies': 21872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1942, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6600, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3528}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6711", 'reported_seq': 6810, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:15.216716+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:15.216716+0000', 'last_peered': '2024-09-10T20:24:15.216716+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:15.216716+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6711, 'log_dups_size': 0, 'ondisk_log_size': 6711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45724832, 'num_objects': 5582, 'num_object_clones': 0, 'num_object_copies': 22328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6709, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6625", 'reported_seq': 8930, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:18.079689+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:18.079689+0000', 'last_peered': '2024-09-10T20:24:18.079689+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:18.079689+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6625, 'log_dups_size': 0, 'ondisk_log_size': 6625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45208736, 'num_objects': 5519, 'num_object_clones': 0, 'num_object_copies': 22076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 798, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6623, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1109, 'num_bytes_recovered': 9052160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4721}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:20.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6611", 'reported_seq': 6723, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:15.978083+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:15.978083+0000', 'last_peered': '2024-09-10T20:24:15.978083+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:15.978083+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6611, 'log_dups_size': 0, 'ondisk_log_size': 6611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45068016, 'num_objects': 5502, 'num_object_clones': 0, 'num_object_copies': 22008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6608, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3568}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6656", 'reported_seq': 6779, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:15.927400+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:15.927400+0000', 'last_peered': '2024-09-10T20:24:15.927400+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:15.927400+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6656, 'log_dups_size': 0, 'ondisk_log_size': 6656, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45128272, 'num_objects': 5509, 'num_object_clones': 0, 'num_object_copies': 22036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1946, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6655, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3564}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6480", 'reported_seq': 6598, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:17.965234+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:17.965234+0000', 'last_peered': '2024-09-10T20:24:17.965234+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:17.965234+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6480, 'log_dups_size': 0, 'ondisk_log_size': 6480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43833936, 'num_objects': 5351, 'num_object_clones': 0, 'num_object_copies': 21404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1934, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6479, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6600", 'reported_seq': 6705, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:15.877100+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:15.877100+0000', 'last_peered': '2024-09-10T20:24:15.877100+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:15.877100+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6600, 'log_dups_size': 0, 'ondisk_log_size': 6600, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45003936, 'num_objects': 5494, 'num_object_clones': 0, 'num_object_copies': 21976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1956, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6598, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3540}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6793", 'reported_seq': 6919, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:18.018079+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:18.018079+0000', 'last_peered': '2024-09-10T20:24:18.018079+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:18.018079+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6793, 'log_dups_size': 0, 'ondisk_log_size': 6793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46331040, 'num_objects': 5656, 'num_object_clones': 0, 'num_object_copies': 22624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1947, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6791, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3711}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:20.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6588", 'reported_seq': 6683, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:15.267754+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:15.267754+0000', 'last_peered': '2024-09-10T20:24:15.267754+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:15.267754+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6588, 'log_dups_size': 0, 'ondisk_log_size': 6588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45356192, 'num_objects': 5537, 'num_object_clones': 0, 'num_object_copies': 22148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1972, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6586, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:20.822 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:21.517 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:21.517 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24209/353016 objects degraded (6.858%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":350,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88254,"data_bytes":722938024,"bytes_used":2772623360,"bytes_avail":383774433280,"bytes_total":386547056640,"degraded_objects":24209,"degraded_total":353016,"degraded_ratio":0.068577628209486249,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":186965,"recovering_keys_per_sec":0,"num_objects_recovered":342,"num_bytes_recovered":2244608,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":252},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:21.547 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:21.548 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:22.166 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-10T20:24:21.976465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":716417192,"num_objects":87458,"num_object_clones":0,"num_object_copies":349832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87458,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94249,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107015,"ondisk_log_size":107015,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742976,"kb_used_data":1335336,"kb_used_omap":14,"kb_used_meta":1407537,"kb_avail":374744384,"statfs":{"total":386547056640,"available":383738249216,"internally_reserved":0,"allocated":1367384064,"data_stored":1366569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1441318454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,8,9,10,0,0,4,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27205632,"num_objects":-3321,"num_object_clones":0,"num_object_copies":-13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3321,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":1687552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005003"},"pg_stats":[{"pgid":"1.f","version":"38'6706","reported_seq":6835,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.010767+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:21.010767+0000","last_peered":"2024-09-10T20:24:21.010767+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:21.010767+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6713","reported_seq":6840,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.891948+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:18.891948+0000","last_peered":"2024-09-10T20:24:18.891948+0000","last_clean":"2024-09-10T20:24:18.891948+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:18.891948+0000","last_undegraded":"2024-09-10T20:24:18.891948+0000","last_fullsized":"2024-09-10T20:24:18.891948+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45375488,"num_objects":5539,"num_object_clones":0,"num_object_copies":22156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3154,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6652","reported_seq":6778,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:17.912318+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:17.912318+0000","last_peered":"2024-09-10T20:24:17.912318+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:17.912318+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6652,"log_dups_size":0,"ondisk_log_size":6652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44956264,"num_objects":5489,"num_object_clones":0,"num_object_copies":21956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6651,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6803","reported_seq":10967,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.869548+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:18.869548+0000","last_peered":"2024-09-10T20:24:18.869548+0000","last_clean":"2024-09-10T20:24:18.869548+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:18.869548+0000","last_undegraded":"2024-09-10T20:24:18.869548+0000","last_fullsized":"2024-09-10T20:24:18.869548+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6803,"log_dups_size":0,"ondisk_log_size":6803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45899776,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1063,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6714","reported_seq":6861,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.891304+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:20.891304+0000","last_peered":"2024-09-10T20:24:20.891304+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:20.891304+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6644","reported_seq":6753,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028105+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:21.028105+0000","last_peered":"2024-09-10T20:24:21.028105+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:21.028105+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44679168,"num_objects":5454,"num_object_clones":0,"num_object_copies":21816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6687","reported_seq":6791,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.263368+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:20.263368+0000","last_peered":"2024-09-10T20:24:20.263368+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:20.263368+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44094624,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3443}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6790","reported_seq":6891,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.353735+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:20.353735+0000","last_peered":"2024-09-10T20:24:20.353735+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:20.353735+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45077664,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6643","reported_seq":10610,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:20.952836+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:20.952836+0000","last_peered":"2024-09-10T20:24:20.952836+0000","last_clean":"2024-09-10T20:24:20.952836+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:20.952836+0000","last_undegraded":"2024-09-10T20:24:20.952836+0000","last_fullsized":"2024-09-10T20:24:20.952836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6643,"log_dups_size":0,"ondisk_log_size":6643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43891280,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6625","reported_seq":8930,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:18.079689+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:18.079689+0000","last_peered":"2024-09-10T20:24:18.079689+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:18.079689+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208736,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6623,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6691","reported_seq":6809,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.886903+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:20.886903+0000","last_peered":"2024-09-10T20:24:20.886903+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:20.886903+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44412656,"num_objects":5422,"num_object_clones":0,"num_object_copies":21688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6744","reported_seq":6884,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028965+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:21.028965+0000","last_peered":"2024-09-10T20:24:21.028965+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:21.028965+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44407376,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6480","reported_seq":6598,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:17.965234+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:17.965234+0000","last_peered":"2024-09-10T20:24:17.965234+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:17.965234+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43833936,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6479,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6679","reported_seq":6797,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.015141+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:21.015141+0000","last_peered":"2024-09-10T20:24:21.015141+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:21.015141+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6679,"log_dups_size":0,"ondisk_log_size":6679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356768,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3467}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6793","reported_seq":6919,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:18.018079+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:18.018079+0000","last_peered":"2024-09-10T20:24:18.018079+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:18.018079+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331040,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6651","reported_seq":6746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.242990+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:20.242990+0000","last_peered":"2024-09-10T20:24:20.242990+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:20.242990+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44840096,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":716417192,"num_objects":87458,"num_object_clones":0,"num_object_copies":349832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87458,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94249,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":9052160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1363292160,"data_stored":1363292160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107015,"ondisk_log_size":107015,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724204,"kb_used_data":349596,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93647636,"statfs":{"total":96636764160,"available":95895179264,"internally_reserved":0,"allocated":357986304,"data_stored":357782624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":383577808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,6,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727084,"kb_used_data":352976,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93644756,"statfs":{"total":96636764160,"available":95892230144,"internally_reserved":0,"allocated":361447424,"data_stored":361243744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,4,3,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566540,"kb_used_data":282168,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93805300,"statfs":{"total":96636764160,"available":96056627200,"internally_reserved":0,"allocated":288940032,"data_stored":288736352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725148,"kb_used_data":350596,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93646692,"statfs":{"total":96636764160,"available":95894212608,"internally_reserved":0,"allocated":359010304,"data_stored":358806624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":383512271},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357687296,"data_stored":357687296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288546816,"data_stored":288546816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360034304,"data_stored":360034304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357023744,"data_stored":357023744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:25.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:25.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:25.770 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:25.793 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-10T20:24:23.977380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":713820328,"num_objects":87141,"num_object_clones":0,"num_object_copies":348564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94566,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1521,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107332,"ondisk_log_size":107332,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742096,"kb_used_data":1330308,"kb_used_omap":14,"kb_used_meta":1411697,"kb_avail":374745264,"statfs":{"total":386547056640,"available":383739150336,"internally_reserved":0,"allocated":1362235392,"data_stored":1361420672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1445578294},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,6,7,7,0,0,2,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22577152,"num_objects":-2756,"num_object_clones":0,"num_object_copies":-11024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2756,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":2523136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005294"},"pg_stats":[{"pgid":"1.f","version":"38'6706","reported_seq":6835,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.010767+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:21.010767+0000","last_peered":"2024-09-10T20:24:21.010767+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:21.010767+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6713","reported_seq":6840,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.891948+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:18.891948+0000","last_peered":"2024-09-10T20:24:18.891948+0000","last_clean":"2024-09-10T20:24:18.891948+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:18.891948+0000","last_undegraded":"2024-09-10T20:24:18.891948+0000","last_fullsized":"2024-09-10T20:24:18.891948+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45375488,"num_objects":5539,"num_object_clones":0,"num_object_copies":22156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3154,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6739","reported_seq":6871,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:22.965519+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:22.965519+0000","last_peered":"2024-09-10T20:24:22.965519+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:22.965519+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243560,"num_objects":5402,"num_object_clones":0,"num_object_copies":21608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6738,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6803","reported_seq":10967,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:18.869548+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:18.869548+0000","last_peered":"2024-09-10T20:24:18.869548+0000","last_clean":"2024-09-10T20:24:18.869548+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:18.869548+0000","last_undegraded":"2024-09-10T20:24:18.869548+0000","last_fullsized":"2024-09-10T20:24:18.869548+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6803,"log_dups_size":0,"ondisk_log_size":6803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45899776,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1063,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6714","reported_seq":6861,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.891304+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:20.891304+0000","last_peered":"2024-09-10T20:24:20.891304+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:20.891304+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6644","reported_seq":6753,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028105+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:21.028105+0000","last_peered":"2024-09-10T20:24:21.028105+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:21.028105+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44679168,"num_objects":5454,"num_object_clones":0,"num_object_copies":21816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6687","reported_seq":6791,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.263368+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:20.263368+0000","last_peered":"2024-09-10T20:24:20.263368+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:20.263368+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44094624,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3443}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6790","reported_seq":6891,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.353735+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:20.353735+0000","last_peered":"2024-09-10T20:24:20.353735+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:20.353735+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45077664,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6643","reported_seq":10610,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:20.952836+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:20.952836+0000","last_peered":"2024-09-10T20:24:20.952836+0000","last_clean":"2024-09-10T20:24:20.952836+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:20.952836+0000","last_undegraded":"2024-09-10T20:24:20.952836+0000","last_fullsized":"2024-09-10T20:24:20.952836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6643,"log_dups_size":0,"ondisk_log_size":6643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43891280,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6705","reported_seq":9214,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.017307+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:23.017307+0000","last_peered":"2024-09-10T20:24:23.017307+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:23.017307+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553376,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1211,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6691","reported_seq":6809,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.886903+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:20.886903+0000","last_peered":"2024-09-10T20:24:20.886903+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:20.886903+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44412656,"num_objects":5422,"num_object_clones":0,"num_object_copies":21688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6744","reported_seq":6884,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028965+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:21.028965+0000","last_peered":"2024-09-10T20:24:21.028965+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:21.028965+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44407376,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6554","reported_seq":6680,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.024383+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:23.024383+0000","last_peered":"2024-09-10T20:24:23.024383+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:23.024383+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43227728,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6679","reported_seq":6797,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.015141+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:21.015141+0000","last_peered":"2024-09-10T20:24:21.015141+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:21.015141+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6679,"log_dups_size":0,"ondisk_log_size":6679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356768,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3467}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6869","reported_seq":6996,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.048223+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:23.048223+0000","last_peered":"2024-09-10T20:24:23.048223+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:23.048223+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708448,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6651","reported_seq":6746,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.242990+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:20.242990+0000","last_peered":"2024-09-10T20:24:20.242990+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:20.242990+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44840096,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":713820328,"num_objects":87141,"num_object_clones":0,"num_object_copies":348564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94566,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1521,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1358077952,"data_stored":1358077952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107332,"ondisk_log_size":107332,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724204,"kb_used_data":349596,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93647636,"statfs":{"total":96636764160,"available":95895179264,"internally_reserved":0,"allocated":357986304,"data_stored":357782624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":383577808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,6,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726204,"kb_used_data":347948,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93645636,"statfs":{"total":96636764160,"available":95893131264,"internally_reserved":0,"allocated":356298752,"data_stored":356095072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566540,"kb_used_data":282168,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93805300,"statfs":{"total":96636764160,"available":96056627200,"internally_reserved":0,"allocated":288940032,"data_stored":288736352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725148,"kb_used_data":350596,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93646692,"statfs":{"total":96636764160,"available":95894212608,"internally_reserved":0,"allocated":359010304,"data_stored":358806624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":383512271},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357687296,"data_stored":357687296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288546816,"data_stored":288546816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354820096,"data_stored":354820096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357023744,"data_stored":357023744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:25.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:26.389 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:26.390 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:26.408 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-10T20:24:25.978051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":710486184,"num_objects":86734,"num_object_clones":0,"num_object_copies":346936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94973,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107739,"ondisk_log_size":107739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748192,"kb_used_data":1323932,"kb_used_omap":14,"kb_used_meta":1424177,"kb_avail":374739168,"statfs":{"total":386547056640,"available":383732908032,"internally_reserved":0,"allocated":1355706368,"data_stored":1354891648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1458357814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,7,8,7,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23494656,"num_objects":-2868,"num_object_clones":0,"num_object_copies":-11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005219"},"pg_stats":[{"pgid":"1.f","version":"38'6706","reported_seq":6835,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.010767+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:21.010767+0000","last_peered":"2024-09-10T20:24:21.010767+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:21.010767+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6807","reported_seq":6934,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:23.927641+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:23.927641+0000","last_peered":"2024-09-10T20:24:23.927641+0000","last_clean":"2024-09-10T20:24:23.927641+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:23.927641+0000","last_undegraded":"2024-09-10T20:24:23.927641+0000","last_fullsized":"2024-09-10T20:24:23.927641+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6807,"log_dups_size":0,"ondisk_log_size":6807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44605440,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3248,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6739","reported_seq":6871,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:22.965519+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:22.965519+0000","last_peered":"2024-09-10T20:24:22.965519+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:22.965519+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243560,"num_objects":5402,"num_object_clones":0,"num_object_copies":21608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6738,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6879","reported_seq":11043,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:23.910679+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:23.910679+0000","last_peered":"2024-09-10T20:24:23.910679+0000","last_clean":"2024-09-10T20:24:23.910679+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:23.910679+0000","last_undegraded":"2024-09-10T20:24:23.910679+0000","last_fullsized":"2024-09-10T20:24:23.910679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6879,"log_dups_size":0,"ondisk_log_size":6879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45277184,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6714","reported_seq":6861,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.891304+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:20.891304+0000","last_peered":"2024-09-10T20:24:20.891304+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:20.891304+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6644","reported_seq":6753,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028105+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:21.028105+0000","last_peered":"2024-09-10T20:24:21.028105+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:21.028105+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44679168,"num_objects":5454,"num_object_clones":0,"num_object_copies":21816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6751","reported_seq":6861,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.276214+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:25.276214+0000","last_peered":"2024-09-10T20:24:25.276214+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:25.276214+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43570336,"num_objects":5319,"num_object_clones":0,"num_object_copies":21276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3382}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6870","reported_seq":6977,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.308264+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:25.308264+0000","last_peered":"2024-09-10T20:24:25.308264+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:25.308264+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44422304,"num_objects":5423,"num_object_clones":0,"num_object_copies":21692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6643","reported_seq":10610,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:20.952836+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:20.952836+0000","last_peered":"2024-09-10T20:24:20.952836+0000","last_clean":"2024-09-10T20:24:20.952836+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:20.952836+0000","last_undegraded":"2024-09-10T20:24:20.952836+0000","last_fullsized":"2024-09-10T20:24:20.952836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6643,"log_dups_size":0,"ondisk_log_size":6643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43891280,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6705","reported_seq":9214,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.017307+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:23.017307+0000","last_peered":"2024-09-10T20:24:23.017307+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:23.017307+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553376,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1211,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6691","reported_seq":6809,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.886903+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:20.886903+0000","last_peered":"2024-09-10T20:24:20.886903+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:20.886903+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44412656,"num_objects":5422,"num_object_clones":0,"num_object_copies":21688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6744","reported_seq":6884,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028965+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:21.028965+0000","last_peered":"2024-09-10T20:24:21.028965+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:21.028965+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44407376,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6554","reported_seq":6680,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.024383+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:23.024383+0000","last_peered":"2024-09-10T20:24:23.024383+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:23.024383+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43227728,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6679","reported_seq":6797,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.015141+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:21.015141+0000","last_peered":"2024-09-10T20:24:21.015141+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:21.015141+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6679,"log_dups_size":0,"ondisk_log_size":6679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356768,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3467}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6869","reported_seq":6996,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.048223+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:23.048223+0000","last_peered":"2024-09-10T20:24:23.048223+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:23.048223+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708448,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6744","reported_seq":6845,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.409976+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:25.409976+0000","last_peered":"2024-09-10T20:24:25.409976+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:25.409976+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44078240,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6742,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":710486184,"num_objects":86734,"num_object_clones":0,"num_object_copies":346936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94973,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1351491584,"data_stored":1351491584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107739,"ondisk_log_size":107739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724204,"kb_used_data":349596,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93647636,"statfs":{"total":96636764160,"available":95895179264,"internally_reserved":0,"allocated":357986304,"data_stored":357782624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":383577808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,6,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726204,"kb_used_data":347948,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93645636,"statfs":{"total":96636764160,"available":95893131264,"internally_reserved":0,"allocated":356298752,"data_stored":356095072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573960,"kb_used_data":281272,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93797880,"statfs":{"total":96636764160,"available":96049029120,"internally_reserved":0,"allocated":288022528,"data_stored":287818848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723824,"kb_used_data":345116,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93648016,"statfs":{"total":96636764160,"available":95895568384,"internally_reserved":0,"allocated":353398784,"data_stored":353195104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":387772111},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352137216,"data_stored":352137216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287510528,"data_stored":287510528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354820096,"data_stored":354820096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357023744,"data_stored":357023744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:26.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:26.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6706", 'reported_seq': 6835, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:21.010767+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:21.010767+0000', 'last_peered': '2024-09-10T20:24:21.010767+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:21.010767+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6706, 'log_dups_size': 0, 'ondisk_log_size': 6706, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44367872, 'num_objects': 5416, 'num_object_clones': 0, 'num_object_copies': 21664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1929, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6706, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3487}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:26.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:26.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6739", 'reported_seq': 6871, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:22.965519+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:22.965519+0000', 'last_peered': '2024-09-10T20:24:22.965519+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:22.965519+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6739, 'log_dups_size': 0, 'ondisk_log_size': 6739, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44243560, 'num_objects': 5402, 'num_object_clones': 0, 'num_object_copies': 21608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1942, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5402, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6738, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3461}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:26.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:26.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6714", 'reported_seq': 6861, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:20.891304+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:20.891304+0000', 'last_peered': '2024-09-10T20:24:20.891304+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:20.891304+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6714, 'log_dups_size': 0, 'ondisk_log_size': 6714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44684448, 'num_objects': 5455, 'num_object_clones': 0, 'num_object_copies': 21820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1941, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6712, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:26.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6644", 'reported_seq': 6753, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:21.028105+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:21.028105+0000', 'last_peered': '2024-09-10T20:24:21.028105+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:21.028105+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6644, 'log_dups_size': 0, 'ondisk_log_size': 6644, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44679168, 'num_objects': 5454, 'num_object_clones': 0, 'num_object_copies': 21816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1911, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6644, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3543}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6751", 'reported_seq': 6861, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:25.276214+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:25.276214+0000', 'last_peered': '2024-09-10T20:24:25.276214+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:25.276214+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6751, 'log_dups_size': 0, 'ondisk_log_size': 6751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43570336, 'num_objects': 5319, 'num_object_clones': 0, 'num_object_copies': 21276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6749, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3382}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6870", 'reported_seq': 6977, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:25.308264+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:25.308264+0000', 'last_peered': '2024-09-10T20:24:25.308264+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:25.308264+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6870, 'log_dups_size': 0, 'ondisk_log_size': 6870, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44422304, 'num_objects': 5423, 'num_object_clones': 0, 'num_object_copies': 21692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6868, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6705", 'reported_seq': 9214, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:23.017307+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:23.017307+0000', 'last_peered': '2024-09-10T20:24:23.017307+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:23.017307+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6705, 'log_dups_size': 0, 'ondisk_log_size': 6705, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44553376, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 696, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6703, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1211, 'num_bytes_recovered': 9887744, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4743}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:26.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6691", 'reported_seq': 6809, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:20.886903+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:20.886903+0000', 'last_peered': '2024-09-10T20:24:20.886903+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:20.886903+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6691, 'log_dups_size': 0, 'ondisk_log_size': 6691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44412656, 'num_objects': 5422, 'num_object_clones': 0, 'num_object_copies': 21688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1934, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5422, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6688, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3491}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6744", 'reported_seq': 6884, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:21.028965+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:21.028965+0000', 'last_peered': '2024-09-10T20:24:21.028965+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:21.028965+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6744, 'log_dups_size': 0, 'ondisk_log_size': 6744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44407376, 'num_objects': 5421, 'num_object_clones': 0, 'num_object_copies': 21684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6743, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6554", 'reported_seq': 6680, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:23.024383+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:23.024383+0000', 'last_peered': '2024-09-10T20:24:23.024383+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:23.024383+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6554, 'log_dups_size': 0, 'ondisk_log_size': 6554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43227728, 'num_objects': 5277, 'num_object_clones': 0, 'num_object_copies': 21108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1930, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6553, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6679", 'reported_seq': 6797, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:21.015141+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:21.015141+0000', 'last_peered': '2024-09-10T20:24:21.015141+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:21.015141+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6679, 'log_dups_size': 0, 'ondisk_log_size': 6679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44356768, 'num_objects': 5415, 'num_object_clones': 0, 'num_object_copies': 21660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6677, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6869", 'reported_seq': 6996, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:23.048223+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:23.048223+0000', 'last_peered': '2024-09-10T20:24:23.048223+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:23.048223+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6869, 'log_dups_size': 0, 'ondisk_log_size': 6869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45708448, 'num_objects': 5580, 'num_object_clones': 0, 'num_object_copies': 22320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6867, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:26.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:26.412 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6744", 'reported_seq': 6845, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:25.409976+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:25.409976+0000', 'last_peered': '2024-09-10T20:24:25.409976+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:25.409976+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6744, 'log_dups_size': 0, 'ondisk_log_size': 6744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44078240, 'num_objects': 5381, 'num_object_clones': 0, 'num_object_copies': 21524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6742, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:26.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:27.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:27.129 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24053/346936 objects degraded (6.933%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":355,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86734,"data_bytes":710486184,"bytes_used":2814148608,"bytes_avail":383732908032,"bytes_total":386547056640,"degraded_objects":24053,"degraded_total":346936,"degraded_ratio":0.069329789932437103,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":138521,"recovering_keys_per_sec":0,"num_objects_recovered":294,"num_bytes_recovered":1662976,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:27.156 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:27.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:27.797 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-10T20:24:25.978051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":710486184,"num_objects":86734,"num_object_clones":0,"num_object_copies":346936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94973,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107739,"ondisk_log_size":107739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748192,"kb_used_data":1323932,"kb_used_omap":14,"kb_used_meta":1424177,"kb_avail":374739168,"statfs":{"total":386547056640,"available":383732908032,"internally_reserved":0,"allocated":1355706368,"data_stored":1354891648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1458357814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,7,8,7,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23494656,"num_objects":-2868,"num_object_clones":0,"num_object_copies":-11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005219"},"pg_stats":[{"pgid":"1.f","version":"38'6706","reported_seq":6835,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.010767+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:21.010767+0000","last_peered":"2024-09-10T20:24:21.010767+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:21.010767+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6807","reported_seq":6934,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:23.927641+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:23.927641+0000","last_peered":"2024-09-10T20:24:23.927641+0000","last_clean":"2024-09-10T20:24:23.927641+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:23.927641+0000","last_undegraded":"2024-09-10T20:24:23.927641+0000","last_fullsized":"2024-09-10T20:24:23.927641+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6807,"log_dups_size":0,"ondisk_log_size":6807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44605440,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3248,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6739","reported_seq":6871,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:22.965519+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:22.965519+0000","last_peered":"2024-09-10T20:24:22.965519+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:22.965519+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243560,"num_objects":5402,"num_object_clones":0,"num_object_copies":21608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6738,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6879","reported_seq":11043,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:23.910679+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:23.910679+0000","last_peered":"2024-09-10T20:24:23.910679+0000","last_clean":"2024-09-10T20:24:23.910679+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:23.910679+0000","last_undegraded":"2024-09-10T20:24:23.910679+0000","last_fullsized":"2024-09-10T20:24:23.910679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6879,"log_dups_size":0,"ondisk_log_size":6879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45277184,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6714","reported_seq":6861,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.891304+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:20.891304+0000","last_peered":"2024-09-10T20:24:20.891304+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:20.891304+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6644","reported_seq":6753,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028105+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:21.028105+0000","last_peered":"2024-09-10T20:24:21.028105+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:21.028105+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44679168,"num_objects":5454,"num_object_clones":0,"num_object_copies":21816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6751","reported_seq":6861,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.276214+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:25.276214+0000","last_peered":"2024-09-10T20:24:25.276214+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:25.276214+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43570336,"num_objects":5319,"num_object_clones":0,"num_object_copies":21276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3382}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6870","reported_seq":6977,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.308264+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:25.308264+0000","last_peered":"2024-09-10T20:24:25.308264+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:25.308264+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44422304,"num_objects":5423,"num_object_clones":0,"num_object_copies":21692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6643","reported_seq":10610,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:20.952836+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:20.952836+0000","last_peered":"2024-09-10T20:24:20.952836+0000","last_clean":"2024-09-10T20:24:20.952836+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:20.952836+0000","last_undegraded":"2024-09-10T20:24:20.952836+0000","last_fullsized":"2024-09-10T20:24:20.952836+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6643,"log_dups_size":0,"ondisk_log_size":6643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43891280,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6705","reported_seq":9214,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.017307+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:23.017307+0000","last_peered":"2024-09-10T20:24:23.017307+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:23.017307+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553376,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1211,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6691","reported_seq":6809,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:20.886903+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:20.886903+0000","last_peered":"2024-09-10T20:24:20.886903+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:20.886903+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44412656,"num_objects":5422,"num_object_clones":0,"num_object_copies":21688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6744","reported_seq":6884,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.028965+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:21.028965+0000","last_peered":"2024-09-10T20:24:21.028965+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:21.028965+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44407376,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6554","reported_seq":6680,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.024383+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:23.024383+0000","last_peered":"2024-09-10T20:24:23.024383+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:23.024383+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43227728,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6679","reported_seq":6797,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:21.015141+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:21.015141+0000","last_peered":"2024-09-10T20:24:21.015141+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:21.015141+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6679,"log_dups_size":0,"ondisk_log_size":6679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356768,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3467}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6869","reported_seq":6996,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:23.048223+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:23.048223+0000","last_peered":"2024-09-10T20:24:23.048223+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:23.048223+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708448,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6744","reported_seq":6845,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.409976+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:25.409976+0000","last_peered":"2024-09-10T20:24:25.409976+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:25.409976+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44078240,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6742,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":710486184,"num_objects":86734,"num_object_clones":0,"num_object_copies":346936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94973,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":9887744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1351491584,"data_stored":1351491584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107739,"ondisk_log_size":107739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724204,"kb_used_data":349596,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93647636,"statfs":{"total":96636764160,"available":95895179264,"internally_reserved":0,"allocated":357986304,"data_stored":357782624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":383577808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,6,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726204,"kb_used_data":347948,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93645636,"statfs":{"total":96636764160,"available":95893131264,"internally_reserved":0,"allocated":356298752,"data_stored":356095072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573960,"kb_used_data":281272,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93797880,"statfs":{"total":96636764160,"available":96049029120,"internally_reserved":0,"allocated":288022528,"data_stored":287818848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723824,"kb_used_data":345116,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93648016,"statfs":{"total":96636764160,"available":95895568384,"internally_reserved":0,"allocated":353398784,"data_stored":353195104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":387772111},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352137216,"data_stored":352137216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287510528,"data_stored":287510528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354820096,"data_stored":354820096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357023744,"data_stored":357023744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:30.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:31.413 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:31.415 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:31.440 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-10T20:24:29.979473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":701360296,"num_objects":85620,"num_object_clones":0,"num_object_copies":342480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96087,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1634,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108853,"ondisk_log_size":108853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749212,"kb_used_data":1311036,"kb_used_omap":14,"kb_used_meta":1438065,"kb_avail":374738148,"statfs":{"total":386547056640,"available":383731863552,"internally_reserved":0,"allocated":1342500864,"data_stored":1341686144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1472579126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,1,19,4,7,0,0,5,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25272320,"num_objects":-3085,"num_object_clones":0,"num_object_copies":-12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":2342912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005274"},"pg_stats":[{"pgid":"1.f","version":"38'6793","reported_seq":6925,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:26.149734+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:26.149734+0000","last_peered":"2024-09-10T20:24:26.149734+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:26.149734+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43655168,"num_objects":5329,"num_object_clones":0,"num_object_copies":21316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3402}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6902","reported_seq":7029,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:28.991211+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:28.991211+0000","last_peered":"2024-09-10T20:24:28.991211+0000","last_clean":"2024-09-10T20:24:28.991211+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:28.991211+0000","last_undegraded":"2024-09-10T20:24:28.991211+0000","last_fullsized":"2024-09-10T20:24:28.991211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6902,"log_dups_size":0,"ondisk_log_size":6902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43827200,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6818","reported_seq":6950,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.038691+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:28.038691+0000","last_peered":"2024-09-10T20:24:28.038691+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:28.038691+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43596392,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6817,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3382}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6959","reported_seq":11123,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:28.968249+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:28.968249+0000","last_peered":"2024-09-10T20:24:28.968249+0000","last_clean":"2024-09-10T20:24:28.968249+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:28.968249+0000","last_undegraded":"2024-09-10T20:24:28.968249+0000","last_fullsized":"2024-09-10T20:24:28.968249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44621824,"num_objects":5447,"num_object_clones":0,"num_object_copies":21788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6785","reported_seq":6936,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.917209+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:25.917209+0000","last_peered":"2024-09-10T20:24:25.917209+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:25.917209+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44102816,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6783,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6748","reported_seq":6859,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:26.033376+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:26.033376+0000","last_peered":"2024-09-10T20:24:26.033376+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:26.033376+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43827200,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6751","reported_seq":6861,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.276214+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:25.276214+0000","last_peered":"2024-09-10T20:24:25.276214+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:25.276214+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43570336,"num_objects":5319,"num_object_clones":0,"num_object_copies":21276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3382}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6870","reported_seq":6977,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.308264+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:25.308264+0000","last_peered":"2024-09-10T20:24:25.308264+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:25.308264+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44422304,"num_objects":5423,"num_object_clones":0,"num_object_copies":21692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6734","reported_seq":10701,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:26.034617+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:26.034617+0000","last_peered":"2024-09-10T20:24:26.034617+0000","last_clean":"2024-09-10T20:24:26.034617+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:26.034617+0000","last_undegraded":"2024-09-10T20:24:26.034617+0000","last_fullsized":"2024-09-10T20:24:26.034617+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6734,"log_dups_size":0,"ondisk_log_size":6734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43145808,"num_objects":5267,"num_object_clones":0,"num_object_copies":21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3287,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6792","reported_seq":9467,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.071225+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:28.071225+0000","last_peered":"2024-09-10T20:24:28.071225+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:28.071225+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43840672,"num_objects":5352,"num_object_clones":0,"num_object_copies":21408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6790,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6775","reported_seq":6905,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:26.057832+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:26.057832+0000","last_peered":"2024-09-10T20:24:26.057832+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:26.057832+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43724528,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6822","reported_seq":6967,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:26.061399+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:26.061399+0000","last_peered":"2024-09-10T20:24:26.061399+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:26.061399+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43768400,"num_objects":5343,"num_object_clones":0,"num_object_copies":21372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3409}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6644","reported_seq":6772,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.093954+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:28.093954+0000","last_peered":"2024-09-10T20:24:28.093954+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:28.093954+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42490448,"num_objects":5187,"num_object_clones":0,"num_object_copies":20748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6767","reported_seq":6899,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:26.061851+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:26.061851+0000","last_peered":"2024-09-10T20:24:26.061851+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:26.061851+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43635872,"num_objects":5327,"num_object_clones":0,"num_object_copies":21308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6949","reported_seq":7079,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.066661+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:28.066661+0000","last_peered":"2024-09-10T20:24:28.066661+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:28.066661+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45053088,"num_objects":5500,"num_object_clones":0,"num_object_copies":22000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6744","reported_seq":6845,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:25.409976+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:25.409976+0000","last_peered":"2024-09-10T20:24:25.409976+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:25.409976+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44078240,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6742,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":701360296,"num_objects":85620,"num_object_clones":0,"num_object_copies":342480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96087,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1634,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339322368,"data_stored":1339322368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108853,"ondisk_log_size":108853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723084,"kb_used_data":344300,"kb_used_omap":4,"kb_used_meta":378747,"kb_avail":93648756,"statfs":{"total":96636764160,"available":95896326144,"internally_reserved":0,"allocated":352563200,"data_stored":352359520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":387837648},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,13,0,1,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741924,"kb_used_data":341456,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93629916,"statfs":{"total":96636764160,"available":95877033984,"internally_reserved":0,"allocated":349650944,"data_stored":349447264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,5,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560380,"kb_used_data":280164,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93811460,"statfs":{"total":96636764160,"available":96062935040,"internally_reserved":0,"allocated":286887936,"data_stored":286684256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723824,"kb_used_data":345116,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93648016,"statfs":{"total":96636764160,"available":95895568384,"internally_reserved":0,"allocated":353398784,"data_stored":353195104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":387772111},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352137216,"data_stored":352137216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286453760,"data_stored":286453760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349331456,"data_stored":349331456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351399936,"data_stored":351399936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:31.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:32.072 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:32.073 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:32.104 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-10T20:24:31.980165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":694593704,"num_objects":84794,"num_object_clones":0,"num_object_copies":339176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84794,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96913,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1654,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109679,"ondisk_log_size":109679,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2783080,"kb_used_data":1300496,"kb_used_omap":14,"kb_used_meta":1482481,"kb_avail":374704280,"statfs":{"total":386547056640,"available":383697182720,"internally_reserved":0,"allocated":1331707904,"data_stored":1330893184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1518061110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,10,15,8,2,0,0,0,1,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28344320,"num_objects":-3460,"num_object_clones":0,"num_object_copies":-13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004423"},"pg_stats":[{"pgid":"1.f","version":"38'6848","reported_seq":6988,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.017235+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:31.017235+0000","last_peered":"2024-09-10T20:24:31.017235+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:31.017235+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43204608,"num_objects":5274,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6902","reported_seq":7029,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:28.991211+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:28.991211+0000","last_peered":"2024-09-10T20:24:28.991211+0000","last_clean":"2024-09-10T20:24:28.991211+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:28.991211+0000","last_undegraded":"2024-09-10T20:24:28.991211+0000","last_fullsized":"2024-09-10T20:24:28.991211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6902,"log_dups_size":0,"ondisk_log_size":6902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43827200,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6818","reported_seq":6950,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.038691+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:28.038691+0000","last_peered":"2024-09-10T20:24:28.038691+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:28.038691+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43596392,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6817,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3382}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6959","reported_seq":11123,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:28.968249+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:28.968249+0000","last_peered":"2024-09-10T20:24:28.968249+0000","last_clean":"2024-09-10T20:24:28.968249+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:28.968249+0000","last_undegraded":"2024-09-10T20:24:28.968249+0000","last_fullsized":"2024-09-10T20:24:28.968249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44621824,"num_objects":5447,"num_object_clones":0,"num_object_copies":21788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6885","reported_seq":7044,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.059492+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:31.059492+0000","last_peered":"2024-09-10T20:24:31.059492+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:31.059492+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43283616,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6827","reported_seq":6942,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.090138+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:31.090138+0000","last_peered":"2024-09-10T20:24:31.090138+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:31.090138+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43180032,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6825","reported_seq":6935,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.373309+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:30.373309+0000","last_peered":"2024-09-10T20:24:30.373309+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:30.373309+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42964128,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6823,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6958","reported_seq":7065,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.336781+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:30.336781+0000","last_peered":"2024-09-10T20:24:30.336781+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:30.336781+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6958,"log_dups_size":0,"ondisk_log_size":6958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43701408,"num_objects":5335,"num_object_clones":0,"num_object_copies":21340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6808","reported_seq":10775,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:31.043051+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:31.043051+0000","last_peered":"2024-09-10T20:24:31.043051+0000","last_clean":"2024-09-10T20:24:31.043051+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:31.043051+0000","last_undegraded":"2024-09-10T20:24:31.043051+0000","last_fullsized":"2024-09-10T20:24:31.043051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42539600,"num_objects":5193,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6792","reported_seq":9467,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.071225+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:28.071225+0000","last_peered":"2024-09-10T20:24:28.071225+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:28.071225+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43840672,"num_objects":5352,"num_object_clones":0,"num_object_copies":21408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6790,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6855","reported_seq":6991,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.022673+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:31.022673+0000","last_peered":"2024-09-10T20:24:31.022673+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:31.022673+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43069168,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6852,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6904","reported_seq":7057,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.927723+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:30.927723+0000","last_peered":"2024-09-10T20:24:30.927723+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:30.927723+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43096656,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3331}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6644","reported_seq":6772,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.093954+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:28.093954+0000","last_peered":"2024-09-10T20:24:28.093954+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:28.093954+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42490448,"num_objects":5187,"num_object_clones":0,"num_object_copies":20748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6865","reported_seq":7001,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.019072+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:31.019072+0000","last_peered":"2024-09-10T20:24:31.019072+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:31.019072+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42833056,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6949","reported_seq":7079,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.066661+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:28.066661+0000","last_peered":"2024-09-10T20:24:28.066661+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:28.066661+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45053088,"num_objects":5500,"num_object_clones":0,"num_object_copies":22000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6840","reported_seq":6941,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.299307+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:30.299307+0000","last_peered":"2024-09-10T20:24:30.299307+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:30.299307+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6840,"log_dups_size":0,"ondisk_log_size":6840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291808,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":694593704,"num_objects":84794,"num_object_clones":0,"num_object_copies":339176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84794,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96913,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1654,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1328627712,"data_stored":1328627712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109679,"ondisk_log_size":109679,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739912,"kb_used_data":338928,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93631928,"statfs":{"total":96636764160,"available":95879094272,"internally_reserved":0,"allocated":347062272,"data_stored":346858592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,1,1,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741924,"kb_used_data":341456,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93629916,"statfs":{"total":96636764160,"available":95877033984,"internally_reserved":0,"allocated":349650944,"data_stored":349447264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,5,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560380,"kb_used_data":280164,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93811460,"statfs":{"total":96636764160,"available":96062935040,"internally_reserved":0,"allocated":286887936,"data_stored":286684256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740864,"kb_used_data":339948,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93630976,"statfs":{"total":96636764160,"available":95878119424,"internally_reserved":0,"allocated":348106752,"data_stored":347903072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346800128,"data_stored":346800128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286453760,"data_stored":286453760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349331456,"data_stored":349331456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346042368,"data_stored":346042368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:32.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:32.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6848", 'reported_seq': 6988, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.017235+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:31.017235+0000', 'last_peered': '2024-09-10T20:24:31.017235+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:31.017235+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6848, 'log_dups_size': 0, 'ondisk_log_size': 6848, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43204608, 'num_objects': 5274, 'num_object_clones': 0, 'num_object_copies': 21096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1923, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6848, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:32.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:32.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6818", 'reported_seq': 6950, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:28.038691+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:28.038691+0000', 'last_peered': '2024-09-10T20:24:28.038691+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:28.038691+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6818, 'log_dups_size': 0, 'ondisk_log_size': 6818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43596392, 'num_objects': 5323, 'num_object_clones': 0, 'num_object_copies': 21292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1942, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5323, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6817, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3382}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:32.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:32.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6885", 'reported_seq': 7044, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.059492+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:31.059492+0000', 'last_peered': '2024-09-10T20:24:31.059492+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:31.059492+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6885, 'log_dups_size': 0, 'ondisk_log_size': 6885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43283616, 'num_objects': 5284, 'num_object_clones': 0, 'num_object_copies': 21136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6883, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:32.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6827", 'reported_seq': 6942, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.090138+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:31.090138+0000', 'last_peered': '2024-09-10T20:24:31.090138+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:31.090138+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6827, 'log_dups_size': 0, 'ondisk_log_size': 6827, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43180032, 'num_objects': 5271, 'num_object_clones': 0, 'num_object_copies': 21084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1908, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6827, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3363}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6825", 'reported_seq': 6935, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:30.373309+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:30.373309+0000', 'last_peered': '2024-09-10T20:24:30.373309+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:30.373309+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6825, 'log_dups_size': 0, 'ondisk_log_size': 6825, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42964128, 'num_objects': 5245, 'num_object_clones': 0, 'num_object_copies': 20980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6823, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3308}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'6958", 'reported_seq': 7065, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:30.336781+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:30.336781+0000', 'last_peered': '2024-09-10T20:24:30.336781+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:30.336781+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6958, 'log_dups_size': 0, 'ondisk_log_size': 6958, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43701408, 'num_objects': 5335, 'num_object_clones': 0, 'num_object_copies': 21340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6956, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6792", 'reported_seq': 9467, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:28.071225+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:28.071225+0000', 'last_peered': '2024-09-10T20:24:28.071225+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:28.071225+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6792, 'log_dups_size': 0, 'ondisk_log_size': 6792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43840672, 'num_objects': 5352, 'num_object_clones': 0, 'num_object_copies': 21408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 613, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6790, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1294, 'num_bytes_recovered': 10567680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4739}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6855", 'reported_seq': 6991, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.022673+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:31.022673+0000', 'last_peered': '2024-09-10T20:24:31.022673+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:31.022673+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6855, 'log_dups_size': 0, 'ondisk_log_size': 6855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43069168, 'num_objects': 5258, 'num_object_clones': 0, 'num_object_copies': 21032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1925, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6852, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3336}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:32.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6904", 'reported_seq': 7057, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:30.927723+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:30.927723+0000', 'last_peered': '2024-09-10T20:24:30.927723+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:30.927723+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6904, 'log_dups_size': 0, 'ondisk_log_size': 6904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43096656, 'num_objects': 5261, 'num_object_clones': 0, 'num_object_copies': 21044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6903, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6644", 'reported_seq': 6772, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:28.093954+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:28.093954+0000', 'last_peered': '2024-09-10T20:24:28.093954+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:28.093954+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6644, 'log_dups_size': 0, 'ondisk_log_size': 6644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42490448, 'num_objects': 5187, 'num_object_clones': 0, 'num_object_copies': 20748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1929, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6643, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6865", 'reported_seq': 7001, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.019072+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:31.019072+0000', 'last_peered': '2024-09-10T20:24:31.019072+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:31.019072+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6865, 'log_dups_size': 0, 'ondisk_log_size': 6865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42833056, 'num_objects': 5229, 'num_object_clones': 0, 'num_object_copies': 20916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1940, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6863, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3291}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'6949", 'reported_seq': 7079, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:28.066661+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:28.066661+0000', 'last_peered': '2024-09-10T20:24:28.066661+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:28.066661+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6949, 'log_dups_size': 0, 'ondisk_log_size': 6949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45053088, 'num_objects': 5500, 'num_object_clones': 0, 'num_object_copies': 22000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6947, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:32.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6840", 'reported_seq': 6941, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:30.299307+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:30.299307+0000', 'last_peered': '2024-09-10T20:24:30.299307+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:30.299307+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6840, 'log_dups_size': 0, 'ondisk_log_size': 6840, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43291808, 'num_objects': 5285, 'num_object_clones': 0, 'num_object_copies': 21140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6838, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:32.107 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:32.813 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:32.813 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23929/339176 objects degraded (7.055%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":361,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84794,"data_bytes":694593704,"bytes_used":2849873920,"bytes_avail":383697182720,"bytes_total":386547056640,"degraded_objects":23929,"degraded_total":339176,"degraded_ratio":0.070550392716465793,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":126246,"recovering_keys_per_sec":0,"num_objects_recovered":280,"num_bytes_recovered":1515520,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":288},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:32.836 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:32.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:33.419 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:33.419 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:33.452 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-10T20:24:31.980165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":694593704,"num_objects":84794,"num_object_clones":0,"num_object_copies":339176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84794,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96913,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1654,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109679,"ondisk_log_size":109679,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2783080,"kb_used_data":1300496,"kb_used_omap":14,"kb_used_meta":1482481,"kb_avail":374704280,"statfs":{"total":386547056640,"available":383697182720,"internally_reserved":0,"allocated":1331707904,"data_stored":1330893184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1518061110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,10,15,8,2,0,0,0,1,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28344320,"num_objects":-3460,"num_object_clones":0,"num_object_copies":-13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004423"},"pg_stats":[{"pgid":"1.f","version":"38'6848","reported_seq":6988,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.017235+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:31.017235+0000","last_peered":"2024-09-10T20:24:31.017235+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:31.017235+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43204608,"num_objects":5274,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6902","reported_seq":7029,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:28.991211+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:28.991211+0000","last_peered":"2024-09-10T20:24:28.991211+0000","last_clean":"2024-09-10T20:24:28.991211+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:28.991211+0000","last_undegraded":"2024-09-10T20:24:28.991211+0000","last_fullsized":"2024-09-10T20:24:28.991211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6902,"log_dups_size":0,"ondisk_log_size":6902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43827200,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6818","reported_seq":6950,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.038691+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:28.038691+0000","last_peered":"2024-09-10T20:24:28.038691+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:28.038691+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43596392,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6817,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3382}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'6959","reported_seq":11123,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:28.968249+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:28.968249+0000","last_peered":"2024-09-10T20:24:28.968249+0000","last_clean":"2024-09-10T20:24:28.968249+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:28.968249+0000","last_undegraded":"2024-09-10T20:24:28.968249+0000","last_fullsized":"2024-09-10T20:24:28.968249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44621824,"num_objects":5447,"num_object_clones":0,"num_object_copies":21788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6885","reported_seq":7044,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.059492+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:31.059492+0000","last_peered":"2024-09-10T20:24:31.059492+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:31.059492+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43283616,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6827","reported_seq":6942,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.090138+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:31.090138+0000","last_peered":"2024-09-10T20:24:31.090138+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:31.090138+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43180032,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6825","reported_seq":6935,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.373309+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:30.373309+0000","last_peered":"2024-09-10T20:24:30.373309+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:30.373309+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42964128,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6823,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'6958","reported_seq":7065,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.336781+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:30.336781+0000","last_peered":"2024-09-10T20:24:30.336781+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:30.336781+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6958,"log_dups_size":0,"ondisk_log_size":6958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43701408,"num_objects":5335,"num_object_clones":0,"num_object_copies":21340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6808","reported_seq":10775,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:31.043051+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:31.043051+0000","last_peered":"2024-09-10T20:24:31.043051+0000","last_clean":"2024-09-10T20:24:31.043051+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:31.043051+0000","last_undegraded":"2024-09-10T20:24:31.043051+0000","last_fullsized":"2024-09-10T20:24:31.043051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42539600,"num_objects":5193,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6792","reported_seq":9467,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.071225+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:28.071225+0000","last_peered":"2024-09-10T20:24:28.071225+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:28.071225+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43840672,"num_objects":5352,"num_object_clones":0,"num_object_copies":21408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6790,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6855","reported_seq":6991,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.022673+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:31.022673+0000","last_peered":"2024-09-10T20:24:31.022673+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:31.022673+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43069168,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6852,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6904","reported_seq":7057,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.927723+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:30.927723+0000","last_peered":"2024-09-10T20:24:30.927723+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:30.927723+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43096656,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3331}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6644","reported_seq":6772,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.093954+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:28.093954+0000","last_peered":"2024-09-10T20:24:28.093954+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:28.093954+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42490448,"num_objects":5187,"num_object_clones":0,"num_object_copies":20748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6865","reported_seq":7001,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.019072+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:31.019072+0000","last_peered":"2024-09-10T20:24:31.019072+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:31.019072+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42833056,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'6949","reported_seq":7079,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:28.066661+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:28.066661+0000","last_peered":"2024-09-10T20:24:28.066661+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:28.066661+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45053088,"num_objects":5500,"num_object_clones":0,"num_object_copies":22000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6840","reported_seq":6941,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.299307+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:30.299307+0000","last_peered":"2024-09-10T20:24:30.299307+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:30.299307+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6840,"log_dups_size":0,"ondisk_log_size":6840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291808,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":694593704,"num_objects":84794,"num_object_clones":0,"num_object_copies":339176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84794,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96913,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1654,"num_bytes_recovered":10567680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1328627712,"data_stored":1328627712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109679,"ondisk_log_size":109679,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739912,"kb_used_data":338928,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93631928,"statfs":{"total":96636764160,"available":95879094272,"internally_reserved":0,"allocated":347062272,"data_stored":346858592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,1,1,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741924,"kb_used_data":341456,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93629916,"statfs":{"total":96636764160,"available":95877033984,"internally_reserved":0,"allocated":349650944,"data_stored":349447264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,5,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560380,"kb_used_data":280164,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93811460,"statfs":{"total":96636764160,"available":96062935040,"internally_reserved":0,"allocated":286887936,"data_stored":286684256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740864,"kb_used_data":339948,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93630976,"statfs":{"total":96636764160,"available":95878119424,"internally_reserved":0,"allocated":348106752,"data_stored":347903072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346800128,"data_stored":346800128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286453760,"data_stored":286453760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349331456,"data_stored":349331456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346042368,"data_stored":346042368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:36.454 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:36.951 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:36.973 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-10T20:24:35.981201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":688851112,"num_objects":84093,"num_object_clones":0,"num_object_copies":336372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97614,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110380,"ondisk_log_size":110380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751620,"kb_used_data":1289832,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374735740,"statfs":{"total":386547056640,"available":383729397760,"internally_reserved":0,"allocated":1320787968,"data_stored":1319973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1496761910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,9,7,7,3,1,0,0,1,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24969216,"num_objects":-3048,"num_object_clones":0,"num_object_copies":-12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003821"},"pg_stats":[{"pgid":"1.f","version":"38'6848","reported_seq":6988,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.017235+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:31.017235+0000","last_peered":"2024-09-10T20:24:31.017235+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:31.017235+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43204608,"num_objects":5274,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6991","reported_seq":7118,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:33.964299+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:33.964299+0000","last_peered":"2024-09-10T20:24:33.964299+0000","last_clean":"2024-09-10T20:24:33.964299+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:33.964299+0000","last_undegraded":"2024-09-10T20:24:33.964299+0000","last_fullsized":"2024-09-10T20:24:33.964299+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43098112,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6899","reported_seq":7033,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.123929+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:33.123929+0000","last_peered":"2024-09-10T20:24:33.123929+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:33.123929+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42932840,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6898,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3302}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7027","reported_seq":11191,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:33.869202+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:33.869202+0000","last_peered":"2024-09-10T20:24:33.869202+0000","last_clean":"2024-09-10T20:24:33.869202+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:33.869202+0000","last_undegraded":"2024-09-10T20:24:33.869202+0000","last_fullsized":"2024-09-10T20:24:33.869202+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7027,"log_dups_size":0,"ondisk_log_size":7027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44064768,"num_objects":5379,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1287,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6885","reported_seq":7044,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.059492+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:31.059492+0000","last_peered":"2024-09-10T20:24:31.059492+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:31.059492+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43283616,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6827","reported_seq":6942,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.090138+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:31.090138+0000","last_peered":"2024-09-10T20:24:31.090138+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:31.090138+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43180032,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6896","reported_seq":7005,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.437387+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:35.437387+0000","last_peered":"2024-09-10T20:24:35.437387+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:35.437387+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382496,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3237}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7044","reported_seq":7153,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.262007+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:35.262007+0000","last_peered":"2024-09-10T20:24:35.262007+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:35.262007+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42996896,"num_objects":5249,"num_object_clones":0,"num_object_copies":20996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6808","reported_seq":10775,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:31.043051+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:31.043051+0000","last_peered":"2024-09-10T20:24:31.043051+0000","last_clean":"2024-09-10T20:24:31.043051+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:31.043051+0000","last_undegraded":"2024-09-10T20:24:31.043051+0000","last_fullsized":"2024-09-10T20:24:31.043051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42539600,"num_objects":5193,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6867","reported_seq":9728,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.107881+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:33.107881+0000","last_peered":"2024-09-10T20:24:33.107881+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:33.107881+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226272,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6865,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1387,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6855","reported_seq":6991,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.022673+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:31.022673+0000","last_peered":"2024-09-10T20:24:31.022673+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:31.022673+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43069168,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6852,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6904","reported_seq":7057,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.927723+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:30.927723+0000","last_peered":"2024-09-10T20:24:30.927723+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:30.927723+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43096656,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3331}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6719","reported_seq":6847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.071661+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:33.071661+0000","last_peered":"2024-09-10T20:24:33.071661+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:33.071661+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41876048,"num_objects":5112,"num_object_clones":0,"num_object_copies":20448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6865","reported_seq":7001,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.019072+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:31.019072+0000","last_peered":"2024-09-10T20:24:31.019072+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:31.019072+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42833056,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7034","reported_seq":7166,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.034900+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:33.034900+0000","last_peered":"2024-09-10T20:24:33.034900+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:33.034900+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7034,"log_dups_size":0,"ondisk_log_size":7034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356768,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6911","reported_seq":7013,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.432255+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:35.432255+0000","last_peered":"2024-09-10T20:24:35.432255+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:35.432255+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6911,"log_dups_size":0,"ondisk_log_size":6911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6909,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":688851112,"num_objects":84093,"num_object_clones":0,"num_object_copies":336372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97614,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1317617664,"data_stored":1317617664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110380,"ondisk_log_size":110380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739912,"kb_used_data":338928,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93631928,"statfs":{"total":96636764160,"available":95879094272,"internally_reserved":0,"allocated":347062272,"data_stored":346858592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,1,1,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724404,"kb_used_data":336412,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93647436,"statfs":{"total":96636764160,"available":95894974464,"internally_reserved":0,"allocated":344485888,"data_stored":344282208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563716,"kb_used_data":279340,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93808124,"statfs":{"total":96636764160,"available":96059518976,"internally_reserved":0,"allocated":286044160,"data_stored":285840480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723588,"kb_used_data":335152,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93648252,"statfs":{"total":96636764160,"available":95895810048,"internally_reserved":0,"allocated":343195648,"data_stored":342991968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341798912,"data_stored":341798912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285634560,"data_stored":285634560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344141824,"data_stored":344141824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346042368,"data_stored":346042368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:36.974 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:37.593 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:37.593 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:37.616 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-10T20:24:35.981201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":688851112,"num_objects":84093,"num_object_clones":0,"num_object_copies":336372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97614,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110380,"ondisk_log_size":110380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751620,"kb_used_data":1289832,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374735740,"statfs":{"total":386547056640,"available":383729397760,"internally_reserved":0,"allocated":1320787968,"data_stored":1319973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1496761910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,9,7,7,3,1,0,0,1,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24969216,"num_objects":-3048,"num_object_clones":0,"num_object_copies":-12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003821"},"pg_stats":[{"pgid":"1.f","version":"38'6848","reported_seq":6988,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.017235+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:31.017235+0000","last_peered":"2024-09-10T20:24:31.017235+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:31.017235+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43204608,"num_objects":5274,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6991","reported_seq":7118,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:33.964299+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:33.964299+0000","last_peered":"2024-09-10T20:24:33.964299+0000","last_clean":"2024-09-10T20:24:33.964299+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:33.964299+0000","last_undegraded":"2024-09-10T20:24:33.964299+0000","last_fullsized":"2024-09-10T20:24:33.964299+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43098112,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6899","reported_seq":7033,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.123929+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:33.123929+0000","last_peered":"2024-09-10T20:24:33.123929+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:33.123929+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42932840,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6898,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3302}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7027","reported_seq":11191,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:33.869202+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:33.869202+0000","last_peered":"2024-09-10T20:24:33.869202+0000","last_clean":"2024-09-10T20:24:33.869202+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:33.869202+0000","last_undegraded":"2024-09-10T20:24:33.869202+0000","last_fullsized":"2024-09-10T20:24:33.869202+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7027,"log_dups_size":0,"ondisk_log_size":7027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44064768,"num_objects":5379,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1287,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6885","reported_seq":7044,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.059492+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:31.059492+0000","last_peered":"2024-09-10T20:24:31.059492+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:31.059492+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43283616,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6827","reported_seq":6942,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.090138+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:31.090138+0000","last_peered":"2024-09-10T20:24:31.090138+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:31.090138+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43180032,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6896","reported_seq":7005,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.437387+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:35.437387+0000","last_peered":"2024-09-10T20:24:35.437387+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:35.437387+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382496,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3237}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7044","reported_seq":7153,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.262007+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:35.262007+0000","last_peered":"2024-09-10T20:24:35.262007+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:35.262007+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42996896,"num_objects":5249,"num_object_clones":0,"num_object_copies":20996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6808","reported_seq":10775,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:31.043051+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:31.043051+0000","last_peered":"2024-09-10T20:24:31.043051+0000","last_clean":"2024-09-10T20:24:31.043051+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:31.043051+0000","last_undegraded":"2024-09-10T20:24:31.043051+0000","last_fullsized":"2024-09-10T20:24:31.043051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42539600,"num_objects":5193,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6867","reported_seq":9728,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.107881+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:33.107881+0000","last_peered":"2024-09-10T20:24:33.107881+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:33.107881+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226272,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6865,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1387,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6855","reported_seq":6991,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.022673+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:31.022673+0000","last_peered":"2024-09-10T20:24:31.022673+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:31.022673+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43069168,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6852,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6904","reported_seq":7057,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:30.927723+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:30.927723+0000","last_peered":"2024-09-10T20:24:30.927723+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:30.927723+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43096656,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3331}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6719","reported_seq":6847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.071661+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:33.071661+0000","last_peered":"2024-09-10T20:24:33.071661+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:33.071661+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41876048,"num_objects":5112,"num_object_clones":0,"num_object_copies":20448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6865","reported_seq":7001,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:31.019072+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:31.019072+0000","last_peered":"2024-09-10T20:24:31.019072+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:31.019072+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42833056,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7034","reported_seq":7166,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.034900+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:33.034900+0000","last_peered":"2024-09-10T20:24:33.034900+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:33.034900+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7034,"log_dups_size":0,"ondisk_log_size":7034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356768,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6911","reported_seq":7013,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.432255+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:35.432255+0000","last_peered":"2024-09-10T20:24:35.432255+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:35.432255+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6911,"log_dups_size":0,"ondisk_log_size":6911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6909,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":688851112,"num_objects":84093,"num_object_clones":0,"num_object_copies":336372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97614,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1317617664,"data_stored":1317617664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110380,"ondisk_log_size":110380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739912,"kb_used_data":338928,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93631928,"statfs":{"total":96636764160,"available":95879094272,"internally_reserved":0,"allocated":347062272,"data_stored":346858592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,1,1,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724404,"kb_used_data":336412,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93647436,"statfs":{"total":96636764160,"available":95894974464,"internally_reserved":0,"allocated":344485888,"data_stored":344282208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563716,"kb_used_data":279340,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93808124,"statfs":{"total":96636764160,"available":96059518976,"internally_reserved":0,"allocated":286044160,"data_stored":285840480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723588,"kb_used_data":335152,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93648252,"statfs":{"total":96636764160,"available":95895810048,"internally_reserved":0,"allocated":343195648,"data_stored":342991968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341798912,"data_stored":341798912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285634560,"data_stored":285634560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344141824,"data_stored":344141824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346042368,"data_stored":346042368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'6848", 'reported_seq': 6988, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.017235+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:31.017235+0000', 'last_peered': '2024-09-10T20:24:31.017235+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:31.017235+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6848, 'log_dups_size': 0, 'ondisk_log_size': 6848, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43204608, 'num_objects': 5274, 'num_object_clones': 0, 'num_object_copies': 21096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1923, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6848, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6899", 'reported_seq': 7033, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:33.123929+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:33.123929+0000', 'last_peered': '2024-09-10T20:24:33.123929+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:33.123929+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6899, 'log_dups_size': 0, 'ondisk_log_size': 6899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42932840, 'num_objects': 5242, 'num_object_clones': 0, 'num_object_copies': 20968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1941, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5242, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6898, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3302}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'6885", 'reported_seq': 7044, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.059492+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:31.059492+0000', 'last_peered': '2024-09-10T20:24:31.059492+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:31.059492+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6885, 'log_dups_size': 0, 'ondisk_log_size': 6885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43283616, 'num_objects': 5284, 'num_object_clones': 0, 'num_object_copies': 21136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6883, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:37.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6827", 'reported_seq': 6942, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.090138+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:31.090138+0000', 'last_peered': '2024-09-10T20:24:31.090138+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:31.090138+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6827, 'log_dups_size': 0, 'ondisk_log_size': 6827, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43180032, 'num_objects': 5271, 'num_object_clones': 0, 'num_object_copies': 21084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1908, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6827, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3363}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6896", 'reported_seq': 7005, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:35.437387+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:35.437387+0000', 'last_peered': '2024-09-10T20:24:35.437387+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:35.437387+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6896, 'log_dups_size': 0, 'ondisk_log_size': 6896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42382496, 'num_objects': 5174, 'num_object_clones': 0, 'num_object_copies': 20696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6894, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3237}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'7044", 'reported_seq': 7153, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:35.262007+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:35.262007+0000', 'last_peered': '2024-09-10T20:24:35.262007+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:35.262007+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7044, 'log_dups_size': 0, 'ondisk_log_size': 7044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42996896, 'num_objects': 5249, 'num_object_clones': 0, 'num_object_copies': 20996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2029, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7042, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3222}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6867", 'reported_seq': 9728, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:33.107881+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:33.107881+0000', 'last_peered': '2024-09-10T20:24:33.107881+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:33.107881+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6867, 'log_dups_size': 0, 'ondisk_log_size': 6867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43226272, 'num_objects': 5277, 'num_object_clones': 0, 'num_object_copies': 21108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 520, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6865, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1387, 'num_bytes_recovered': 11329536, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4757}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'6855", 'reported_seq': 6991, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.022673+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:31.022673+0000', 'last_peered': '2024-09-10T20:24:31.022673+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:31.022673+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6855, 'log_dups_size': 0, 'ondisk_log_size': 6855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43069168, 'num_objects': 5258, 'num_object_clones': 0, 'num_object_copies': 21032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1925, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6852, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3336}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:37.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'6904", 'reported_seq': 7057, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:30.927723+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:30.927723+0000', 'last_peered': '2024-09-10T20:24:30.927723+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:30.927723+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6904, 'log_dups_size': 0, 'ondisk_log_size': 6904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43096656, 'num_objects': 5261, 'num_object_clones': 0, 'num_object_copies': 21044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6903, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6719", 'reported_seq': 6847, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:33.071661+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:33.071661+0000', 'last_peered': '2024-09-10T20:24:33.071661+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:33.071661+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6719, 'log_dups_size': 0, 'ondisk_log_size': 6719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41876048, 'num_objects': 5112, 'num_object_clones': 0, 'num_object_copies': 20448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1929, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6718, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'6865", 'reported_seq': 7001, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:31.019072+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:31.019072+0000', 'last_peered': '2024-09-10T20:24:31.019072+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:31.019072+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6865, 'log_dups_size': 0, 'ondisk_log_size': 6865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42833056, 'num_objects': 5229, 'num_object_clones': 0, 'num_object_copies': 20916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1940, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6863, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3291}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'7034", 'reported_seq': 7166, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:33.034900+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:33.034900+0000', 'last_peered': '2024-09-10T20:24:33.034900+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:33.034900+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7034, 'log_dups_size': 0, 'ondisk_log_size': 7034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44356768, 'num_objects': 5415, 'num_object_clones': 0, 'num_object_copies': 21660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1944, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7032, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3473}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:37.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6911", 'reported_seq': 7013, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:35.432255+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:35.432255+0000', 'last_peered': '2024-09-10T20:24:35.432255+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:35.432255+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6911, 'log_dups_size': 0, 'ondisk_log_size': 6911, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42710176, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6909, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3248}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:37.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:38.352 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:38.353 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23832/336372 objects degraded (7.085%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":367,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84093,"data_bytes":688851112,"bytes_used":2817658880,"bytes_avail":383729397760,"bytes_total":386547056640,"degraded_objects":23832,"degraded_total":336372,"degraded_ratio":0.070850130212978493,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":120111,"recovering_keys_per_sec":0,"num_objects_recovered":230,"num_bytes_recovered":1441792,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:38.375 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:38.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:38.977 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:38.978 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:39.000 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-10T20:24:37.981661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":684435624,"num_objects":83554,"num_object_clones":0,"num_object_copies":334216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98153,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110919,"ondisk_log_size":110919,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738448,"kb_used_data":1284940,"kb_used_omap":14,"kb_used_meta":1453361,"kb_avail":374748912,"statfs":{"total":386547056640,"available":383742885888,"internally_reserved":0,"allocated":1315778560,"data_stored":1314963840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1488242230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,5,2,7,4,7,2,0,0,0,1,0,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26050560,"num_objects":-3180,"num_object_clones":0,"num_object_copies":-12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003609"},"pg_stats":[{"pgid":"1.f","version":"38'6925","reported_seq":7066,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:36.100511+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:36.100511+0000","last_peered":"2024-09-10T20:24:36.100511+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:36.100511+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6925,"log_dups_size":0,"ondisk_log_size":6925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42573824,"num_objects":5197,"num_object_clones":0,"num_object_copies":20788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6925,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'6991","reported_seq":7118,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:33.964299+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:33.964299+0000","last_peered":"2024-09-10T20:24:33.964299+0000","last_clean":"2024-09-10T20:24:33.964299+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:33.964299+0000","last_undegraded":"2024-09-10T20:24:33.964299+0000","last_fullsized":"2024-09-10T20:24:33.964299+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43098112,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6899","reported_seq":7033,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.123929+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:33.123929+0000","last_peered":"2024-09-10T20:24:33.123929+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:33.123929+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42932840,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6898,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3302}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7027","reported_seq":11191,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:33.869202+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:33.869202+0000","last_peered":"2024-09-10T20:24:33.869202+0000","last_clean":"2024-09-10T20:24:33.869202+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:33.869202+0000","last_undegraded":"2024-09-10T20:24:33.869202+0000","last_fullsized":"2024-09-10T20:24:33.869202+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7027,"log_dups_size":0,"ondisk_log_size":7027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44064768,"num_objects":5379,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1287,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'6958","reported_seq":7123,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:36.031810+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:36.031810+0000","last_peered":"2024-09-10T20:24:36.031810+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:36.031810+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6958,"log_dups_size":0,"ondisk_log_size":6958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42685600,"num_objects":5211,"num_object_clones":0,"num_object_copies":20844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6922","reported_seq":7041,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.829273+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:35.829273+0000","last_peered":"2024-09-10T20:24:35.829273+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:35.829273+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6922,"log_dups_size":0,"ondisk_log_size":6922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42401792,"num_objects":5176,"num_object_clones":0,"num_object_copies":20704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6922,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6896","reported_seq":7005,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.437387+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:35.437387+0000","last_peered":"2024-09-10T20:24:35.437387+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:35.437387+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382496,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3237}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7044","reported_seq":7153,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.262007+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:35.262007+0000","last_peered":"2024-09-10T20:24:35.262007+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:35.262007+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42996896,"num_objects":5249,"num_object_clones":0,"num_object_copies":20996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6882","reported_seq":10849,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:36.005796+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:36.005796+0000","last_peered":"2024-09-10T20:24:36.005796+0000","last_clean":"2024-09-10T20:24:36.005796+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:36.005796+0000","last_undegraded":"2024-09-10T20:24:36.005796+0000","last_fullsized":"2024-09-10T20:24:36.005796+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41933392,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6867","reported_seq":9728,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.107881+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:33.107881+0000","last_peered":"2024-09-10T20:24:33.107881+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:33.107881+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226272,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6865,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1387,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'6940","reported_seq":7086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:36.062260+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:36.062260+0000","last_peered":"2024-09-10T20:24:36.062260+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:36.062260+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6940,"log_dups_size":0,"ondisk_log_size":6940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42372848,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6937,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'6967","reported_seq":7120,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.939709+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:35.939709+0000","last_peered":"2024-09-10T20:24:35.939709+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:35.939709+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6967,"log_dups_size":0,"ondisk_log_size":6967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42580560,"num_objects":5198,"num_object_clones":0,"num_object_copies":20792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6966,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3268}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6719","reported_seq":6847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.071661+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:33.071661+0000","last_peered":"2024-09-10T20:24:33.071661+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:33.071661+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41876048,"num_objects":5112,"num_object_clones":0,"num_object_copies":20448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'6937","reported_seq":7069,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:36.003785+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:36.003785+0000","last_peered":"2024-09-10T20:24:36.003785+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:36.003785+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6937,"log_dups_size":0,"ondisk_log_size":6937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42243232,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6935,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3219}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7034","reported_seq":7166,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:33.034900+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:33.034900+0000","last_peered":"2024-09-10T20:24:33.034900+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:33.034900+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7034,"log_dups_size":0,"ondisk_log_size":7034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356768,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6911","reported_seq":7013,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:35.432255+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:35.432255+0000","last_peered":"2024-09-10T20:24:35.432255+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:35.432255+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6911,"log_dups_size":0,"ondisk_log_size":6911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6909,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":684435624,"num_objects":83554,"num_object_clones":0,"num_object_copies":334216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98153,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":11329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1312563200,"data_stored":1312563200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110919,"ondisk_log_size":110919,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117058,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726740,"kb_used_data":334036,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93645100,"statfs":{"total":96636764160,"available":95892582400,"internally_reserved":0,"allocated":342052864,"data_stored":341849184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,1,1,5,0,0,0,0,0,0,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724404,"kb_used_data":336412,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93647436,"statfs":{"total":96636764160,"available":95894974464,"internally_reserved":0,"allocated":344485888,"data_stored":344282208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563716,"kb_used_data":279340,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93808124,"statfs":{"total":96636764160,"available":96059518976,"internally_reserved":0,"allocated":286044160,"data_stored":285840480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723588,"kb_used_data":335152,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93648252,"statfs":{"total":96636764160,"available":95895810048,"internally_reserved":0,"allocated":343195648,"data_stored":342991968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341798912,"data_stored":341798912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285634560,"data_stored":285634560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344141824,"data_stored":344141824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340987904,"data_stored":340987904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:42.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:42.637 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:42.665 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-10T20:24:41.983092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":674400424,"num_objects":82329,"num_object_clones":0,"num_object_copies":329316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99378,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":12247040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112144,"ondisk_log_size":112144,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747580,"kb_used_data":1269176,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374739780,"statfs":{"total":386547056640,"available":383733534720,"internally_reserved":0,"allocated":1299636224,"data_stored":1298821504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1513801270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,6,17,8,9,4,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26959872,"num_objects":-3291,"num_object_clones":0,"num_object_copies":-13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"1.f","version":"38'7005","reported_seq":7152,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.048713+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:41.048713+0000","last_peered":"2024-09-10T20:24:41.048713+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:41.048713+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41918464,"num_objects":5117,"num_object_clones":0,"num_object_copies":20468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7066","reported_seq":7193,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:38.984700+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:38.984700+0000","last_peered":"2024-09-10T20:24:38.984700+0000","last_clean":"2024-09-10T20:24:38.984700+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:38.984700+0000","last_undegraded":"2024-09-10T20:24:38.984700+0000","last_fullsized":"2024-09-10T20:24:38.984700+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7066,"log_dups_size":0,"ondisk_log_size":7066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42483712,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6964","reported_seq":7100,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:38.033370+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:38.033370+0000","last_peered":"2024-09-10T20:24:38.033370+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:38.033370+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42400360,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6963,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7105","reported_seq":11269,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:39.063871+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:39.063871+0000","last_peered":"2024-09-10T20:24:39.063871+0000","last_clean":"2024-09-10T20:24:39.063871+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:39.063871+0000","last_undegraded":"2024-09-10T20:24:39.063871+0000","last_fullsized":"2024-09-10T20:24:39.063871+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7105,"log_dups_size":0,"ondisk_log_size":7105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43425792,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7041","reported_seq":7236,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.080366+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:41.080366+0000","last_peered":"2024-09-10T20:24:41.080366+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:41.080366+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005664,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6991","reported_seq":7116,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.042226+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:41.042226+0000","last_peered":"2024-09-10T20:24:41.042226+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:41.042226+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41836544,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6991,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3204}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6975","reported_seq":7103,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.344744+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:40.344744+0000","last_peered":"2024-09-10T20:24:40.344744+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:40.344744+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6975,"log_dups_size":0,"ondisk_log_size":6975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41735328,"num_objects":5095,"num_object_clones":0,"num_object_copies":20380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6973,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7113","reported_seq":7226,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.376857+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:40.376857+0000","last_peered":"2024-09-10T20:24:40.376857+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:40.376857+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7113,"log_dups_size":0,"ondisk_log_size":7113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42431648,"num_objects":5180,"num_object_clones":0,"num_object_copies":20720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6966","reported_seq":10933,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:41.123917+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:41.123917+0000","last_peered":"2024-09-10T20:24:41.123917+0000","last_clean":"2024-09-10T20:24:41.123917+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:41.123917+0000","last_undegraded":"2024-09-10T20:24:41.123917+0000","last_fullsized":"2024-09-10T20:24:41.123917+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41245264,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6936","reported_seq":10020,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:38.172019+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:38.172019+0000","last_peered":"2024-09-10T20:24:38.172019+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:38.172019+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42661024,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":12247040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4800}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7016","reported_seq":7164,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.102286+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:41.102286+0000","last_peered":"2024-09-10T20:24:41.102286+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:41.102286+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41750256,"num_objects":5097,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7047","reported_seq":7214,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.115777+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:41.115777+0000","last_peered":"2024-09-10T20:24:41.115777+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:41.115777+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41925200,"num_objects":5118,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6792","reported_seq":6926,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:37.947704+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:37.947704+0000","last_peered":"2024-09-10T20:24:37.947704+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:37.947704+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41278032,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7017","reported_seq":7161,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.082786+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:41.082786+0000","last_peered":"2024-09-10T20:24:41.082786+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:41.082786+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587872,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7015,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3142}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7112","reported_seq":7250,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:38.091012+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:38.091012+0000","last_peered":"2024-09-10T20:24:38.091012+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:38.091012+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7112,"log_dups_size":0,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717792,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3398}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6998","reported_seq":7111,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.278736+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:40.278736+0000","last_peered":"2024-09-10T20:24:40.278736+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:40.278736+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":674400424,"num_objects":82329,"num_object_clones":0,"num_object_copies":329316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99378,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":12247040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1296379904,"data_stored":1296379904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112144,"ondisk_log_size":112144,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117059,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725868,"kb_used_data":329052,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93645972,"statfs":{"total":96636764160,"available":95893475328,"internally_reserved":0,"allocated":336949248,"data_stored":336745568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727884,"kb_used_data":331580,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93643956,"statfs":{"total":96636764160,"available":95891410944,"internally_reserved":0,"allocated":339537920,"data_stored":339334240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,3,3,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567116,"kb_used_data":278588,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93804724,"statfs":{"total":96636764160,"available":96056037376,"internally_reserved":0,"allocated":285274112,"data_stored":285070432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726712,"kb_used_data":329956,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93645128,"statfs":{"total":96636764160,"available":95892611072,"internally_reserved":0,"allocated":337874944,"data_stored":337671264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336633856,"data_stored":336633856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284815360,"data_stored":284815360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339107840,"data_stored":339107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335822848,"data_stored":335822848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:42.666 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:43.263 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:43.263 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:43.286 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-10T20:24:41.983092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":674400424,"num_objects":82329,"num_object_clones":0,"num_object_copies":329316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99378,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":12247040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112144,"ondisk_log_size":112144,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747580,"kb_used_data":1269176,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374739780,"statfs":{"total":386547056640,"available":383733534720,"internally_reserved":0,"allocated":1299636224,"data_stored":1298821504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1513801270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,6,17,8,9,4,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26959872,"num_objects":-3291,"num_object_clones":0,"num_object_copies":-13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"1.f","version":"38'7005","reported_seq":7152,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.048713+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:41.048713+0000","last_peered":"2024-09-10T20:24:41.048713+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:41.048713+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41918464,"num_objects":5117,"num_object_clones":0,"num_object_copies":20468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7066","reported_seq":7193,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:38.984700+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:38.984700+0000","last_peered":"2024-09-10T20:24:38.984700+0000","last_clean":"2024-09-10T20:24:38.984700+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:38.984700+0000","last_undegraded":"2024-09-10T20:24:38.984700+0000","last_fullsized":"2024-09-10T20:24:38.984700+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7066,"log_dups_size":0,"ondisk_log_size":7066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42483712,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'6964","reported_seq":7100,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:38.033370+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:38.033370+0000","last_peered":"2024-09-10T20:24:38.033370+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:38.033370+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42400360,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6963,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7105","reported_seq":11269,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:39.063871+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:39.063871+0000","last_peered":"2024-09-10T20:24:39.063871+0000","last_clean":"2024-09-10T20:24:39.063871+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:39.063871+0000","last_undegraded":"2024-09-10T20:24:39.063871+0000","last_fullsized":"2024-09-10T20:24:39.063871+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7105,"log_dups_size":0,"ondisk_log_size":7105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43425792,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7041","reported_seq":7236,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.080366+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:41.080366+0000","last_peered":"2024-09-10T20:24:41.080366+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:41.080366+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005664,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6991","reported_seq":7116,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.042226+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:41.042226+0000","last_peered":"2024-09-10T20:24:41.042226+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:41.042226+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41836544,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6991,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3204}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6975","reported_seq":7103,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.344744+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:40.344744+0000","last_peered":"2024-09-10T20:24:40.344744+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:40.344744+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6975,"log_dups_size":0,"ondisk_log_size":6975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41735328,"num_objects":5095,"num_object_clones":0,"num_object_copies":20380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6973,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7113","reported_seq":7226,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.376857+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:40.376857+0000","last_peered":"2024-09-10T20:24:40.376857+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:40.376857+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7113,"log_dups_size":0,"ondisk_log_size":7113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42431648,"num_objects":5180,"num_object_clones":0,"num_object_copies":20720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6966","reported_seq":10933,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:41.123917+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:41.123917+0000","last_peered":"2024-09-10T20:24:41.123917+0000","last_clean":"2024-09-10T20:24:41.123917+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:41.123917+0000","last_undegraded":"2024-09-10T20:24:41.123917+0000","last_fullsized":"2024-09-10T20:24:41.123917+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41245264,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'6936","reported_seq":10020,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:38.172019+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:38.172019+0000","last_peered":"2024-09-10T20:24:38.172019+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:38.172019+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42661024,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":12247040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4800}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7016","reported_seq":7164,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.102286+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:41.102286+0000","last_peered":"2024-09-10T20:24:41.102286+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:41.102286+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41750256,"num_objects":5097,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7047","reported_seq":7214,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.115777+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:41.115777+0000","last_peered":"2024-09-10T20:24:41.115777+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:41.115777+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41925200,"num_objects":5118,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6792","reported_seq":6926,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:37.947704+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:37.947704+0000","last_peered":"2024-09-10T20:24:37.947704+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:37.947704+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41278032,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7017","reported_seq":7161,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.082786+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:41.082786+0000","last_peered":"2024-09-10T20:24:41.082786+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:41.082786+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587872,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7015,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3142}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7112","reported_seq":7250,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:38.091012+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:38.091012+0000","last_peered":"2024-09-10T20:24:38.091012+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:38.091012+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7112,"log_dups_size":0,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717792,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3398}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6998","reported_seq":7111,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.278736+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:40.278736+0000","last_peered":"2024-09-10T20:24:40.278736+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:40.278736+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":674400424,"num_objects":82329,"num_object_clones":0,"num_object_copies":329316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99378,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":12247040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1296379904,"data_stored":1296379904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112144,"ondisk_log_size":112144,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117059,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725868,"kb_used_data":329052,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93645972,"statfs":{"total":96636764160,"available":95893475328,"internally_reserved":0,"allocated":336949248,"data_stored":336745568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727884,"kb_used_data":331580,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93643956,"statfs":{"total":96636764160,"available":95891410944,"internally_reserved":0,"allocated":339537920,"data_stored":339334240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,3,3,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567116,"kb_used_data":278588,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93804724,"statfs":{"total":96636764160,"available":96056037376,"internally_reserved":0,"allocated":285274112,"data_stored":285070432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726712,"kb_used_data":329956,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93645128,"statfs":{"total":96636764160,"available":95892611072,"internally_reserved":0,"allocated":337874944,"data_stored":337671264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336633856,"data_stored":336633856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284815360,"data_stored":284815360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339107840,"data_stored":339107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335822848,"data_stored":335822848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:43.286 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'7005", 'reported_seq': 7152, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:41.048713+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:41.048713+0000', 'last_peered': '2024-09-10T20:24:41.048713+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:41.048713+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7005, 'log_dups_size': 0, 'ondisk_log_size': 7005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41918464, 'num_objects': 5117, 'num_object_clones': 0, 'num_object_copies': 20468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1920, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7005, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3197}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'6964", 'reported_seq': 7100, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:38.033370+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:38.033370+0000', 'last_peered': '2024-09-10T20:24:38.033370+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:38.033370+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6964, 'log_dups_size': 0, 'ondisk_log_size': 6964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42400360, 'num_objects': 5177, 'num_object_clones': 0, 'num_object_copies': 20708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1940, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5177, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6963, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3238}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'7041", 'reported_seq': 7236, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:41.080366+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:41.080366+0000', 'last_peered': '2024-09-10T20:24:41.080366+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:41.080366+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7041, 'log_dups_size': 0, 'ondisk_log_size': 7041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42005664, 'num_objects': 5128, 'num_object_clones': 0, 'num_object_copies': 20512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1917, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7039, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3213}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'6991", 'reported_seq': 7116, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:41.042226+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:41.042226+0000', 'last_peered': '2024-09-10T20:24:41.042226+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:41.042226+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6991, 'log_dups_size': 0, 'ondisk_log_size': 6991, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41836544, 'num_objects': 5107, 'num_object_clones': 0, 'num_object_copies': 20428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1903, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6991, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3204}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'6975", 'reported_seq': 7103, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:40.344744+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:40.344744+0000', 'last_peered': '2024-09-10T20:24:40.344744+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:40.344744+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6975, 'log_dups_size': 0, 'ondisk_log_size': 6975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41735328, 'num_objects': 5095, 'num_object_clones': 0, 'num_object_copies': 20380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1930, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6973, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3167}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'7113", 'reported_seq': 7226, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:40.376857+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:40.376857+0000', 'last_peered': '2024-09-10T20:24:40.376857+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:40.376857+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7113, 'log_dups_size': 0, 'ondisk_log_size': 7113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42431648, 'num_objects': 5180, 'num_object_clones': 0, 'num_object_copies': 20720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2027, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7111, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'6936", 'reported_seq': 10020, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:38.172019+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:38.172019+0000', 'last_peered': '2024-09-10T20:24:38.172019+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:38.172019+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6936, 'log_dups_size': 0, 'ondisk_log_size': 6936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42661024, 'num_objects': 5208, 'num_object_clones': 0, 'num_object_copies': 20832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 408, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6934, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1499, 'num_bytes_recovered': 12247040, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4800}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'7016", 'reported_seq': 7164, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:41.102286+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:41.102286+0000', 'last_peered': '2024-09-10T20:24:41.102286+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:41.102286+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7016, 'log_dups_size': 0, 'ondisk_log_size': 7016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41750256, 'num_objects': 5097, 'num_object_clones': 0, 'num_object_copies': 20388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1917, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7013, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'7047", 'reported_seq': 7214, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:41.115777+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:41.115777+0000', 'last_peered': '2024-09-10T20:24:41.115777+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:41.115777+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7047, 'log_dups_size': 0, 'ondisk_log_size': 7047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41925200, 'num_objects': 5118, 'num_object_clones': 0, 'num_object_copies': 20472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1924, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7046, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3195}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6792", 'reported_seq': 6926, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:37.947704+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:37.947704+0000', 'last_peered': '2024-09-10T20:24:37.947704+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:37.947704+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6792, 'log_dups_size': 0, 'ondisk_log_size': 6792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41278032, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 20156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1926, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6791, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'7017", 'reported_seq': 7161, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:41.082786+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:41.082786+0000', 'last_peered': '2024-09-10T20:24:41.082786+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:41.082786+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7017, 'log_dups_size': 0, 'ondisk_log_size': 7017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41587872, 'num_objects': 5077, 'num_object_clones': 0, 'num_object_copies': 20308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7015, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3142}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'7112", 'reported_seq': 7250, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:38.091012+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:38.091012+0000', 'last_peered': '2024-09-10T20:24:38.091012+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:38.091012+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7112, 'log_dups_size': 0, 'ondisk_log_size': 7112, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43717792, 'num_objects': 5337, 'num_object_clones': 0, 'num_object_copies': 21348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1941, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7110, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:43.289 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:43.289 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'6998", 'reported_seq': 7111, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:40.278736+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:40.278736+0000', 'last_peered': '2024-09-10T20:24:40.278736+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:40.278736+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6998, 'log_dups_size': 0, 'ondisk_log_size': 6998, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41997472, 'num_objects': 5127, 'num_object_clones': 0, 'num_object_copies': 20508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6996, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3166}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:43.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:44.056 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:44.057 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23653/329316 objects degraded (7.182%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":372,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82329,"data_bytes":674400424,"bytes_used":2813521920,"bytes_avail":383733534720,"bytes_total":386547056640,"degraded_objects":23653,"degraded_total":329316,"degraded_ratio":0.071824630446136836,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":139904,"recovering_keys_per_sec":0,"num_objects_recovered":296,"num_bytes_recovered":1679360,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":274},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:44.095 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:44.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:45.300 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:45.300 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:45.323 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-10T20:24:43.983714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671221928,"num_objects":81941,"num_object_clones":0,"num_object_copies":327764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99766,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112532,"ondisk_log_size":112532,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2746456,"kb_used_data":1263884,"kb_used_omap":14,"kb_used_meta":1482481,"kb_avail":374740904,"statfs":{"total":386547056640,"available":383734685696,"internally_reserved":0,"allocated":1294217216,"data_stored":1293402496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1518061110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,5,14,8,7,4,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23371776,"num_objects":-2853,"num_object_clones":0,"num_object_copies":-11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":2441216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003548"},"pg_stats":[{"pgid":"1.f","version":"38'7005","reported_seq":7152,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.048713+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:41.048713+0000","last_peered":"2024-09-10T20:24:41.048713+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:41.048713+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41918464,"num_objects":5117,"num_object_clones":0,"num_object_copies":20468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7066","reported_seq":7193,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:38.984700+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:38.984700+0000","last_peered":"2024-09-10T20:24:38.984700+0000","last_clean":"2024-09-10T20:24:38.984700+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:38.984700+0000","last_undegraded":"2024-09-10T20:24:38.984700+0000","last_fullsized":"2024-09-10T20:24:38.984700+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7066,"log_dups_size":0,"ondisk_log_size":7066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42483712,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7056","reported_seq":7204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.173571+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:43.173571+0000","last_peered":"2024-09-10T20:24:43.173571+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:43.173571+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41646696,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7055,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3152}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7105","reported_seq":11269,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:39.063871+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:39.063871+0000","last_peered":"2024-09-10T20:24:39.063871+0000","last_clean":"2024-09-10T20:24:39.063871+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:39.063871+0000","last_undegraded":"2024-09-10T20:24:39.063871+0000","last_fullsized":"2024-09-10T20:24:39.063871+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7105,"log_dups_size":0,"ondisk_log_size":7105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43425792,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7041","reported_seq":7236,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.080366+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:41.080366+0000","last_peered":"2024-09-10T20:24:41.080366+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:41.080366+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005664,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'6991","reported_seq":7116,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.042226+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:41.042226+0000","last_peered":"2024-09-10T20:24:41.042226+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:41.042226+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41836544,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6991,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3204}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'6975","reported_seq":7103,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.344744+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:40.344744+0000","last_peered":"2024-09-10T20:24:40.344744+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:40.344744+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6975,"log_dups_size":0,"ondisk_log_size":6975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41735328,"num_objects":5095,"num_object_clones":0,"num_object_copies":20380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6973,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7113","reported_seq":7226,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.376857+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:40.376857+0000","last_peered":"2024-09-10T20:24:40.376857+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:40.376857+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7113,"log_dups_size":0,"ondisk_log_size":7113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42431648,"num_objects":5180,"num_object_clones":0,"num_object_copies":20720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'6966","reported_seq":10933,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:41.123917+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:41.123917+0000","last_peered":"2024-09-10T20:24:41.123917+0000","last_clean":"2024-09-10T20:24:41.123917+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:41.123917+0000","last_undegraded":"2024-09-10T20:24:41.123917+0000","last_fullsized":"2024-09-10T20:24:41.123917+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41245264,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7040","reported_seq":10311,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.234664+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:43.234664+0000","last_peered":"2024-09-10T20:24:43.234664+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:43.234664+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7040,"log_dups_size":0,"ondisk_log_size":7040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41809056,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7038,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1592,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7016","reported_seq":7164,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.102286+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:41.102286+0000","last_peered":"2024-09-10T20:24:41.102286+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:41.102286+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41750256,"num_objects":5097,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7047","reported_seq":7214,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.115777+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:41.115777+0000","last_peered":"2024-09-10T20:24:41.115777+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:41.115777+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41925200,"num_objects":5118,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6888","reported_seq":7030,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.155091+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:43.155091+0000","last_peered":"2024-09-10T20:24:43.155091+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:43.155091+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40491600,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3022}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7017","reported_seq":7161,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:41.082786+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:41.082786+0000","last_peered":"2024-09-10T20:24:41.082786+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:41.082786+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587872,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7015,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3142}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7208","reported_seq":7346,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.183772+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:43.183772+0000","last_peered":"2024-09-10T20:24:43.183772+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:43.183772+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7208,"log_dups_size":0,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42931360,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7206,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'6998","reported_seq":7111,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:40.278736+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:40.278736+0000","last_peered":"2024-09-10T20:24:40.278736+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:40.278736+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":671221928,"num_objects":81941,"num_object_clones":0,"num_object_copies":327764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99766,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1291018240,"data_stored":1291018240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112532,"ondisk_log_size":112532,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117059,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725868,"kb_used_data":329052,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93645972,"statfs":{"total":96636764160,"available":95893475328,"internally_reserved":0,"allocated":336949248,"data_stored":336745568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726760,"kb_used_data":326288,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93645080,"statfs":{"total":96636764160,"available":95892561920,"internally_reserved":0,"allocated":334118912,"data_stored":333915232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,3,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567116,"kb_used_data":278588,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93804724,"statfs":{"total":96636764160,"available":96056037376,"internally_reserved":0,"allocated":285274112,"data_stored":285070432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726712,"kb_used_data":329956,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93645128,"statfs":{"total":96636764160,"available":95892611072,"internally_reserved":0,"allocated":337874944,"data_stored":337671264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336633856,"data_stored":336633856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284815360,"data_stored":284815360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333746176,"data_stored":333746176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335822848,"data_stored":335822848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:48.325 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:48.908 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:48.930 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-10T20:24:47.985465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663275688,"num_objects":80971,"num_object_clones":0,"num_object_copies":323884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80971,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100736,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113502,"ondisk_log_size":113502,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747688,"kb_used_data":1252640,"kb_used_omap":14,"kb_used_meta":1494961,"kb_avail":374739672,"statfs":{"total":386547056640,"available":383733424128,"internally_reserved":0,"allocated":1282703360,"data_stored":1281888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1530840630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,11,0,7,10,7,3,1,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25575424,"num_objects":-3122,"num_object_clones":0,"num_object_copies":-12488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004263"},"pg_stats":[{"pgid":"1.f","version":"38'7087","reported_seq":7244,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.260799+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:46.260799+0000","last_peered":"2024-09-10T20:24:46.260799+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:46.260799+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41246720,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7139","reported_seq":7265,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:44.087886+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:44.087886+0000","last_peered":"2024-09-10T20:24:44.087886+0000","last_clean":"2024-09-10T20:24:44.087886+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:44.087886+0000","last_undegraded":"2024-09-10T20:24:44.087886+0000","last_fullsized":"2024-09-10T20:24:44.087886+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7139,"log_dups_size":0,"ondisk_log_size":7139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41885696,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3580,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7056","reported_seq":7204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.173571+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:43.173571+0000","last_peered":"2024-09-10T20:24:43.173571+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:43.173571+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41646696,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7055,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3152}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7184","reported_seq":11347,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:44.058283+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:44.058283+0000","last_peered":"2024-09-10T20:24:44.058283+0000","last_clean":"2024-09-10T20:24:44.058283+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:44.058283+0000","last_undegraded":"2024-09-10T20:24:44.058283+0000","last_fullsized":"2024-09-10T20:24:44.058283+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7184,"log_dups_size":0,"ondisk_log_size":7184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7113","reported_seq":7309,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.151994+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:46.151994+0000","last_peered":"2024-09-10T20:24:46.151994+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:46.151994+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7113,"log_dups_size":0,"ondisk_log_size":7113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415840,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7076","reported_seq":7211,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.215293+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:46.215293+0000","last_peered":"2024-09-10T20:24:46.215293+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:46.215293+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41140224,"num_objects":5022,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7076,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7043","reported_seq":7173,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.418376+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:45.418376+0000","last_peered":"2024-09-10T20:24:45.418376+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:45.418376+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41178272,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7200","reported_seq":7312,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.473441+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:45.473441+0000","last_peered":"2024-09-10T20:24:45.473441+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:45.473441+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41718944,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7044","reported_seq":11011,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:46.223412+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:46.223412+0000","last_peered":"2024-09-10T20:24:46.223412+0000","last_clean":"2024-09-10T20:24:46.223412+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:46.223412+0000","last_undegraded":"2024-09-10T20:24:46.223412+0000","last_fullsized":"2024-09-10T20:24:46.223412+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40606288,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7040","reported_seq":10311,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.234664+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:43.234664+0000","last_peered":"2024-09-10T20:24:43.234664+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:43.234664+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7040,"log_dups_size":0,"ondisk_log_size":7040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41809056,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7038,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1592,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7093","reported_seq":7257,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.166148+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:46.166148+0000","last_peered":"2024-09-10T20:24:46.166148+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:46.166148+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41119472,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7132","reported_seq":7307,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.197732+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:46.197732+0000","last_peered":"2024-09-10T20:24:46.197732+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:46.197732+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41228880,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6888","reported_seq":7030,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.155091+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:43.155091+0000","last_peered":"2024-09-10T20:24:43.155091+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:43.155091+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40491600,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3022}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7117","reported_seq":7265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.157711+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:46.157711+0000","last_peered":"2024-09-10T20:24:46.157711+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:46.157711+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7115,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3044}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7208","reported_seq":7346,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.183772+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:43.183772+0000","last_peered":"2024-09-10T20:24:43.183772+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:43.183772+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7208,"log_dups_size":0,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42931360,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7206,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7082","reported_seq":7201,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.270674+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:45.270674+0000","last_peered":"2024-09-10T20:24:45.270674+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:45.270674+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41309344,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":663275688,"num_objects":80971,"num_object_clones":0,"num_object_copies":323884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80971,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100736,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1279266816,"data_stored":1279266816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113502,"ondisk_log_size":113502,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117060,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724964,"kb_used_data":323988,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93646876,"statfs":{"total":96636764160,"available":95894401024,"internally_reserved":0,"allocated":331763712,"data_stored":331560032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,5,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726760,"kb_used_data":326288,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93645080,"statfs":{"total":96636764160,"available":95892561920,"internally_reserved":0,"allocated":334118912,"data_stored":333915232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,3,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570308,"kb_used_data":277620,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93801532,"statfs":{"total":96636764160,"available":96052768768,"internally_reserved":0,"allocated":284282880,"data_stored":284079200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725656,"kb_used_data":324744,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93646184,"statfs":{"total":96636764160,"available":95893692416,"internally_reserved":0,"allocated":332537856,"data_stored":332334176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331333632,"data_stored":331333632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":283791360,"data_stored":283791360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333746176,"data_stored":333746176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330395648,"data_stored":330395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:48.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:49.497 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:49.522 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-10T20:24:47.985465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663275688,"num_objects":80971,"num_object_clones":0,"num_object_copies":323884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80971,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100736,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113502,"ondisk_log_size":113502,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747688,"kb_used_data":1252640,"kb_used_omap":14,"kb_used_meta":1494961,"kb_avail":374739672,"statfs":{"total":386547056640,"available":383733424128,"internally_reserved":0,"allocated":1282703360,"data_stored":1281888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1530840630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,11,0,7,10,7,3,1,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25575424,"num_objects":-3122,"num_object_clones":0,"num_object_copies":-12488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004263"},"pg_stats":[{"pgid":"1.f","version":"38'7087","reported_seq":7244,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.260799+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:46.260799+0000","last_peered":"2024-09-10T20:24:46.260799+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:46.260799+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41246720,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7139","reported_seq":7265,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:44.087886+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:44.087886+0000","last_peered":"2024-09-10T20:24:44.087886+0000","last_clean":"2024-09-10T20:24:44.087886+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:44.087886+0000","last_undegraded":"2024-09-10T20:24:44.087886+0000","last_fullsized":"2024-09-10T20:24:44.087886+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7139,"log_dups_size":0,"ondisk_log_size":7139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41885696,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3580,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7056","reported_seq":7204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.173571+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:43.173571+0000","last_peered":"2024-09-10T20:24:43.173571+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:43.173571+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41646696,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7055,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3152}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7184","reported_seq":11347,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:44.058283+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:44.058283+0000","last_peered":"2024-09-10T20:24:44.058283+0000","last_clean":"2024-09-10T20:24:44.058283+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:44.058283+0000","last_undegraded":"2024-09-10T20:24:44.058283+0000","last_fullsized":"2024-09-10T20:24:44.058283+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7184,"log_dups_size":0,"ondisk_log_size":7184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7113","reported_seq":7309,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.151994+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:46.151994+0000","last_peered":"2024-09-10T20:24:46.151994+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:46.151994+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7113,"log_dups_size":0,"ondisk_log_size":7113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415840,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7076","reported_seq":7211,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.215293+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:46.215293+0000","last_peered":"2024-09-10T20:24:46.215293+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:46.215293+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41140224,"num_objects":5022,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7076,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7043","reported_seq":7173,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.418376+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:45.418376+0000","last_peered":"2024-09-10T20:24:45.418376+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:45.418376+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41178272,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7200","reported_seq":7312,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.473441+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:45.473441+0000","last_peered":"2024-09-10T20:24:45.473441+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:45.473441+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41718944,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7044","reported_seq":11011,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:46.223412+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:46.223412+0000","last_peered":"2024-09-10T20:24:46.223412+0000","last_clean":"2024-09-10T20:24:46.223412+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:46.223412+0000","last_undegraded":"2024-09-10T20:24:46.223412+0000","last_fullsized":"2024-09-10T20:24:46.223412+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40606288,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7040","reported_seq":10311,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.234664+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:43.234664+0000","last_peered":"2024-09-10T20:24:43.234664+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:43.234664+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7040,"log_dups_size":0,"ondisk_log_size":7040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41809056,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7038,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1592,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7093","reported_seq":7257,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.166148+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:46.166148+0000","last_peered":"2024-09-10T20:24:46.166148+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:46.166148+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41119472,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7132","reported_seq":7307,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.197732+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:46.197732+0000","last_peered":"2024-09-10T20:24:46.197732+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:46.197732+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41228880,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6888","reported_seq":7030,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.155091+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:43.155091+0000","last_peered":"2024-09-10T20:24:43.155091+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:43.155091+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40491600,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3022}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7117","reported_seq":7265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.157711+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:46.157711+0000","last_peered":"2024-09-10T20:24:46.157711+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:46.157711+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7115,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3044}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7208","reported_seq":7346,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:43.183772+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:43.183772+0000","last_peered":"2024-09-10T20:24:43.183772+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:43.183772+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7208,"log_dups_size":0,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42931360,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7206,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7082","reported_seq":7201,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.270674+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:45.270674+0000","last_peered":"2024-09-10T20:24:45.270674+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:45.270674+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41309344,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":663275688,"num_objects":80971,"num_object_clones":0,"num_object_copies":323884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80971,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100736,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":13008896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1279266816,"data_stored":1279266816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113502,"ondisk_log_size":113502,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117060,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724964,"kb_used_data":323988,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93646876,"statfs":{"total":96636764160,"available":95894401024,"internally_reserved":0,"allocated":331763712,"data_stored":331560032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,5,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726760,"kb_used_data":326288,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93645080,"statfs":{"total":96636764160,"available":95892561920,"internally_reserved":0,"allocated":334118912,"data_stored":333915232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,3,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570308,"kb_used_data":277620,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93801532,"statfs":{"total":96636764160,"available":96052768768,"internally_reserved":0,"allocated":284282880,"data_stored":284079200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725656,"kb_used_data":324744,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93646184,"statfs":{"total":96636764160,"available":95893692416,"internally_reserved":0,"allocated":332537856,"data_stored":332334176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331333632,"data_stored":331333632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":283791360,"data_stored":283791360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333746176,"data_stored":333746176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330395648,"data_stored":330395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:49.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'7087", 'reported_seq': 7244, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:46.260799+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:46.260799+0000', 'last_peered': '2024-09-10T20:24:46.260799+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:46.260799+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7087, 'log_dups_size': 0, 'ondisk_log_size': 7087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41246720, 'num_objects': 5035, 'num_object_clones': 0, 'num_object_copies': 20140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1915, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7087, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'7056", 'reported_seq': 7204, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:43.173571+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:43.173571+0000', 'last_peered': '2024-09-10T20:24:43.173571+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:43.173571+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7056, 'log_dups_size': 0, 'ondisk_log_size': 7056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41646696, 'num_objects': 5085, 'num_object_clones': 0, 'num_object_copies': 20340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1934, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5085, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7055, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3152}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'7113", 'reported_seq': 7309, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:46.151994+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:46.151994+0000', 'last_peered': '2024-09-10T20:24:46.151994+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:46.151994+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7113, 'log_dups_size': 0, 'ondisk_log_size': 7113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41415840, 'num_objects': 5056, 'num_object_clones': 0, 'num_object_copies': 20224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7111, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3142}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'7076", 'reported_seq': 7211, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:46.215293+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:46.215293+0000', 'last_peered': '2024-09-10T20:24:46.215293+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:46.215293+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7076, 'log_dups_size': 0, 'ondisk_log_size': 7076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41140224, 'num_objects': 5022, 'num_object_clones': 0, 'num_object_copies': 20088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1898, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7076, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'7043", 'reported_seq': 7173, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:45.418376+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:45.418376+0000', 'last_peered': '2024-09-10T20:24:45.418376+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:45.418376+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7043, 'log_dups_size': 0, 'ondisk_log_size': 7043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41178272, 'num_objects': 5027, 'num_object_clones': 0, 'num_object_copies': 20108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1929, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7041, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3100}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:49.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'7200", 'reported_seq': 7312, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:45.473441+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:45.473441+0000', 'last_peered': '2024-09-10T20:24:45.473441+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:45.473441+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7200, 'log_dups_size': 0, 'ondisk_log_size': 7200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41718944, 'num_objects': 5093, 'num_object_clones': 0, 'num_object_copies': 20372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2026, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3069}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'7040", 'reported_seq': 10311, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:43.234664+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:43.234664+0000', 'last_peered': '2024-09-10T20:24:43.234664+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:43.234664+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7040, 'log_dups_size': 0, 'ondisk_log_size': 7040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41809056, 'num_objects': 5104, 'num_object_clones': 0, 'num_object_copies': 20416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7038, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1592, 'num_bytes_recovered': 13008896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4789}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'7093", 'reported_seq': 7257, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:46.166148+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:46.166148+0000', 'last_peered': '2024-09-10T20:24:46.166148+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:46.166148+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7093, 'log_dups_size': 0, 'ondisk_log_size': 7093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41119472, 'num_objects': 5020, 'num_object_clones': 0, 'num_object_copies': 20080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1911, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7090, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3112}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'7132", 'reported_seq': 7307, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:46.197732+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:46.197732+0000', 'last_peered': '2024-09-10T20:24:46.197732+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:46.197732+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7132, 'log_dups_size': 0, 'ondisk_log_size': 7132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41228880, 'num_objects': 5033, 'num_object_clones': 0, 'num_object_copies': 20132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1920, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7131, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:49.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'6888", 'reported_seq': 7030, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:43.155091+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:43.155091+0000', 'last_peered': '2024-09-10T20:24:43.155091+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:43.155091+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 6888, 'log_dups_size': 0, 'ondisk_log_size': 6888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40491600, 'num_objects': 4943, 'num_object_clones': 0, 'num_object_copies': 19772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1922, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6887, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3022}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:49.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:49.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'7117", 'reported_seq': 7265, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:46.157711+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:46.157711+0000', 'last_peered': '2024-09-10T20:24:46.157711+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:46.157711+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7117, 'log_dups_size': 0, 'ondisk_log_size': 7117, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40768672, 'num_objects': 4977, 'num_object_clones': 0, 'num_object_copies': 19908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7115, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:49.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:49.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'7208", 'reported_seq': 7346, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:43.183772+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:43.183772+0000', 'last_peered': '2024-09-10T20:24:43.183772+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:43.183772+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7208, 'log_dups_size': 0, 'ondisk_log_size': 7208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42931360, 'num_objects': 5241, 'num_object_clones': 0, 'num_object_copies': 20964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1941, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7206, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:49.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:49.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'7082", 'reported_seq': 7201, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:45.270674+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:45.270674+0000', 'last_peered': '2024-09-10T20:24:45.270674+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:45.270674+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7082, 'log_dups_size': 0, 'ondisk_log_size': 7082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41309344, 'num_objects': 5043, 'num_object_clones': 0, 'num_object_copies': 20172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7080, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3085}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:49.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:50.227 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23443/321976 objects degraded (7.281%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":378,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80494,"data_bytes":659368104,"bytes_used":2781556736,"bytes_avail":383765499904,"bytes_total":386547056640,"degraded_objects":23443,"degraded_total":321976,"degraded_ratio":0.072809774641588193,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":190390,"recovering_keys_per_sec":0,"num_objects_recovered":379,"num_bytes_recovered":2285568,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:50.248 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:50.249 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:50.940 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:50.965 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-10T20:24:49.986319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659368104,"num_objects":80494,"num_object_clones":0,"num_object_copies":321976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101213,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":13615104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113979,"ondisk_log_size":113979,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716364,"kb_used_data":1246284,"kb_used_omap":14,"kb_used_meta":1470001,"kb_avail":374770996,"statfs":{"total":386547056640,"available":383765499904,"internally_reserved":0,"allocated":1276194816,"data_stored":1275380096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1505281590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,3,5,16,10,3,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25067520,"num_objects":-3060,"num_object_clones":0,"num_object_copies":-12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3060,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":2285568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004658"},"pg_stats":[{"pgid":"1.f","version":"38'7087","reported_seq":7244,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.260799+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:46.260799+0000","last_peered":"2024-09-10T20:24:46.260799+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:46.260799+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41246720,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7210","reported_seq":7337,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:49.112134+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:49.112134+0000","last_peered":"2024-09-10T20:24:49.112134+0000","last_clean":"2024-09-10T20:24:49.112134+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:49.112134+0000","last_undegraded":"2024-09-10T20:24:49.112134+0000","last_fullsized":"2024-09-10T20:24:49.112134+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41304064,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7136","reported_seq":7288,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:48.234695+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:48.234695+0000","last_peered":"2024-09-10T20:24:48.234695+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:48.234695+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7136,"log_dups_size":0,"ondisk_log_size":7136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40991336,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7135,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3074}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7264","reported_seq":11428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:49.129058+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:49.129058+0000","last_peered":"2024-09-10T20:24:49.129058+0000","last_clean":"2024-09-10T20:24:49.129058+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:49.129058+0000","last_undegraded":"2024-09-10T20:24:49.129058+0000","last_fullsized":"2024-09-10T20:24:49.129058+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42123264,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7113","reported_seq":7309,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.151994+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:46.151994+0000","last_peered":"2024-09-10T20:24:46.151994+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:46.151994+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7113,"log_dups_size":0,"ondisk_log_size":7113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415840,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7076","reported_seq":7211,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.215293+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:46.215293+0000","last_peered":"2024-09-10T20:24:46.215293+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:46.215293+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41140224,"num_objects":5022,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7076,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7043","reported_seq":7173,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.418376+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:45.418376+0000","last_peered":"2024-09-10T20:24:45.418376+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:45.418376+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41178272,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7200","reported_seq":7312,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.473441+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:45.473441+0000","last_peered":"2024-09-10T20:24:45.473441+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:45.473441+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41718944,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7044","reported_seq":11011,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:46.223412+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:46.223412+0000","last_peered":"2024-09-10T20:24:46.223412+0000","last_clean":"2024-09-10T20:24:46.223412+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:46.223412+0000","last_undegraded":"2024-09-10T20:24:46.223412+0000","last_fullsized":"2024-09-10T20:24:46.223412+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40606288,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7127","reported_seq":10546,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:48.236296+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:48.236296+0000","last_peered":"2024-09-10T20:24:48.236296+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:48.236296+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7127,"log_dups_size":0,"ondisk_log_size":7127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7125,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":13615104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7093","reported_seq":7257,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.166148+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:46.166148+0000","last_peered":"2024-09-10T20:24:46.166148+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:46.166148+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41119472,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7132","reported_seq":7307,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.197732+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:46.197732+0000","last_peered":"2024-09-10T20:24:46.197732+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:46.197732+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41228880,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'6975","reported_seq":7118,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:48.232601+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:48.232601+0000","last_peered":"2024-09-10T20:24:48.232601+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:48.232601+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":6975,"log_dups_size":0,"ondisk_log_size":6975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39778896,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6974,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7117","reported_seq":7265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:46.157711+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:46.157711+0000","last_peered":"2024-09-10T20:24:46.157711+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:46.157711+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7115,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3044}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7280","reported_seq":7422,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:48.061959+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:48.061959+0000","last_peered":"2024-09-10T20:24:48.061959+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:48.061959+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341536,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7082","reported_seq":7201,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:45.270674+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:45.270674+0000","last_peered":"2024-09-10T20:24:45.270674+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:45.270674+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41309344,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":659368104,"num_objects":80494,"num_object_clones":0,"num_object_copies":321976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101213,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":13615104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1272750080,"data_stored":1272750080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113979,"ondisk_log_size":113979,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117060,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724964,"kb_used_data":323988,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93646876,"statfs":{"total":96636764160,"available":95894401024,"internally_reserved":0,"allocated":331763712,"data_stored":331560032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,5,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708892,"kb_used_data":320908,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93662948,"statfs":{"total":96636764160,"available":95910858752,"internally_reserved":0,"allocated":328609792,"data_stored":328406112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556852,"kb_used_data":276644,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93814988,"statfs":{"total":96636764160,"available":96066547712,"internally_reserved":0,"allocated":283283456,"data_stored":283079776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725656,"kb_used_data":324744,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93646184,"statfs":{"total":96636764160,"available":95893692416,"internally_reserved":0,"allocated":332537856,"data_stored":332334176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331333632,"data_stored":331333632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282832896,"data_stored":282832896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":328187904,"data_stored":328187904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330395648,"data_stored":330395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:53.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:54.600 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:54.625 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-10T20:24:53.987501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650258600,"num_objects":79382,"num_object_clones":0,"num_object_copies":317528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102325,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115091,"ondisk_log_size":115091,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2696940,"kb_used_data":1230988,"kb_used_omap":14,"kb_used_meta":1465841,"kb_avail":374790420,"statfs":{"total":386547056640,"available":383785390080,"internally_reserved":0,"allocated":1260531712,"data_stored":1259716992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1501021750},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,7,11,13,7,0,0,0,0,4,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24141824,"num_objects":-2947,"num_object_clones":0,"num_object_copies":-11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004409"},"pg_stats":[{"pgid":"1.f","version":"38'7167","reported_seq":7332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.992797+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:50.992797+0000","last_peered":"2024-09-10T20:24:50.992797+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:50.992797+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7167,"log_dups_size":0,"ondisk_log_size":7167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40591360,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7210","reported_seq":7337,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:49.112134+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:49.112134+0000","last_peered":"2024-09-10T20:24:49.112134+0000","last_clean":"2024-09-10T20:24:49.112134+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:49.112134+0000","last_undegraded":"2024-09-10T20:24:49.112134+0000","last_fullsized":"2024-09-10T20:24:49.112134+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41304064,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7221","reported_seq":7372,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.297673+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:53.297673+0000","last_peered":"2024-09-10T20:24:53.297673+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:53.297673+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40295016,"num_objects":4920,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7220,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2989}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7264","reported_seq":11428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:49.129058+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:49.129058+0000","last_peered":"2024-09-10T20:24:49.129058+0000","last_clean":"2024-09-10T20:24:49.129058+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:49.129058+0000","last_undegraded":"2024-09-10T20:24:49.129058+0000","last_fullsized":"2024-09-10T20:24:49.129058+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42123264,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7190","reported_seq":7387,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.280915+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:51.280915+0000","last_peered":"2024-09-10T20:24:51.280915+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:51.280915+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7190,"log_dups_size":0,"ondisk_log_size":7190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7147","reported_seq":7292,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.295259+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:51.295259+0000","last_peered":"2024-09-10T20:24:51.295259+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:51.295259+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7147,"log_dups_size":0,"ondisk_log_size":7147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40558592,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7128","reported_seq":7260,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.420125+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:50.420125+0000","last_peered":"2024-09-10T20:24:50.420125+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:50.420125+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7128,"log_dups_size":0,"ondisk_log_size":7128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481952,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3016}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7289","reported_seq":7404,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.285671+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:50.285671+0000","last_peered":"2024-09-10T20:24:50.285671+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:50.285671+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7289,"log_dups_size":0,"ondisk_log_size":7289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989856,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7114","reported_seq":11081,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:51.154556+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:51.154556+0000","last_peered":"2024-09-10T20:24:51.154556+0000","last_clean":"2024-09-10T20:24:51.154556+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:51.154556+0000","last_undegraded":"2024-09-10T20:24:51.154556+0000","last_fullsized":"2024-09-10T20:24:51.154556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40032848,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3667,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7209","reported_seq":10804,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.286656+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:53.286656+0000","last_peered":"2024-09-10T20:24:53.286656+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:53.286656+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7209,"log_dups_size":0,"ondisk_log_size":7209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40424608,"num_objects":4935,"num_object_clones":0,"num_object_copies":19740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7174","reported_seq":7348,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.337928+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:51.337928+0000","last_peered":"2024-09-10T20:24:51.337928+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:51.337928+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7174,"log_dups_size":0,"ondisk_log_size":7174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40455920,"num_objects":4939,"num_object_clones":0,"num_object_copies":19756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7171,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7225","reported_seq":7416,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.963950+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:50.963950+0000","last_peered":"2024-09-10T20:24:50.963950+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:50.963950+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40467024,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7224,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'7053","reported_seq":7199,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.237437+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:53.237437+0000","last_peered":"2024-09-10T20:24:53.237437+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:53.237437+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39139920,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7052,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7185","reported_seq":7337,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.258204+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:51.258204+0000","last_peered":"2024-09-10T20:24:51.258204+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:51.258204+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7185,"log_dups_size":0,"ondisk_log_size":7185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7363","reported_seq":7509,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.119125+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:53.119125+0000","last_peered":"2024-09-10T20:24:53.119125+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:53.119125+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41661600,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3151}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7152","reported_seq":7271,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.159471+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:50.159471+0000","last_peered":"2024-09-10T20:24:50.159471+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:50.159471+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735904,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":650258600,"num_objects":79382,"num_object_clones":0,"num_object_copies":317528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102325,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1257099264,"data_stored":1257099264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115091,"ondisk_log_size":115091,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117061,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724004,"kb_used_data":318840,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93647836,"statfs":{"total":96636764160,"available":95895384064,"internally_reserved":0,"allocated":326492160,"data_stored":326288480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,5,5,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708256,"kb_used_data":316108,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93663584,"statfs":{"total":96636764160,"available":95911510016,"internally_reserved":0,"allocated":323694592,"data_stored":323490912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556852,"kb_used_data":276644,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93814988,"statfs":{"total":96636764160,"available":96066547712,"internally_reserved":0,"allocated":283283456,"data_stored":283079776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707828,"kb_used_data":319396,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93664012,"statfs":{"total":96636764160,"available":95911948288,"internally_reserved":0,"allocated":327061504,"data_stored":326857824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326049792,"data_stored":326049792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282832896,"data_stored":282832896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323092480,"data_stored":323092480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325124096,"data_stored":325124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:54.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:55.123 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:55.124 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-10T20:24:53.987501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650258600,"num_objects":79382,"num_object_clones":0,"num_object_copies":317528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102325,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115091,"ondisk_log_size":115091,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2696940,"kb_used_data":1230988,"kb_used_omap":14,"kb_used_meta":1465841,"kb_avail":374790420,"statfs":{"total":386547056640,"available":383785390080,"internally_reserved":0,"allocated":1260531712,"data_stored":1259716992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1501021750},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,7,11,13,7,0,0,0,0,4,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24141824,"num_objects":-2947,"num_object_clones":0,"num_object_copies":-11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004409"},"pg_stats":[{"pgid":"1.f","version":"38'7167","reported_seq":7332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.992797+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:50.992797+0000","last_peered":"2024-09-10T20:24:50.992797+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:50.992797+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7167,"log_dups_size":0,"ondisk_log_size":7167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40591360,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7210","reported_seq":7337,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:49.112134+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:49.112134+0000","last_peered":"2024-09-10T20:24:49.112134+0000","last_clean":"2024-09-10T20:24:49.112134+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:49.112134+0000","last_undegraded":"2024-09-10T20:24:49.112134+0000","last_fullsized":"2024-09-10T20:24:49.112134+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41304064,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7221","reported_seq":7372,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.297673+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:53.297673+0000","last_peered":"2024-09-10T20:24:53.297673+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:53.297673+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40295016,"num_objects":4920,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7220,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2989}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7264","reported_seq":11428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:49.129058+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:49.129058+0000","last_peered":"2024-09-10T20:24:49.129058+0000","last_clean":"2024-09-10T20:24:49.129058+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:49.129058+0000","last_undegraded":"2024-09-10T20:24:49.129058+0000","last_fullsized":"2024-09-10T20:24:49.129058+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42123264,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7190","reported_seq":7387,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.280915+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:51.280915+0000","last_peered":"2024-09-10T20:24:51.280915+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:51.280915+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7190,"log_dups_size":0,"ondisk_log_size":7190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7147","reported_seq":7292,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.295259+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:51.295259+0000","last_peered":"2024-09-10T20:24:51.295259+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:51.295259+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7147,"log_dups_size":0,"ondisk_log_size":7147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40558592,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7128","reported_seq":7260,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.420125+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:50.420125+0000","last_peered":"2024-09-10T20:24:50.420125+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:50.420125+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7128,"log_dups_size":0,"ondisk_log_size":7128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481952,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3016}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7289","reported_seq":7404,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.285671+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:50.285671+0000","last_peered":"2024-09-10T20:24:50.285671+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:50.285671+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7289,"log_dups_size":0,"ondisk_log_size":7289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989856,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7114","reported_seq":11081,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:51.154556+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:51.154556+0000","last_peered":"2024-09-10T20:24:51.154556+0000","last_clean":"2024-09-10T20:24:51.154556+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:51.154556+0000","last_undegraded":"2024-09-10T20:24:51.154556+0000","last_fullsized":"2024-09-10T20:24:51.154556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40032848,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3667,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7209","reported_seq":10804,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.286656+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:53.286656+0000","last_peered":"2024-09-10T20:24:53.286656+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:53.286656+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7209,"log_dups_size":0,"ondisk_log_size":7209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40424608,"num_objects":4935,"num_object_clones":0,"num_object_copies":19740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7174","reported_seq":7348,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.337928+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:51.337928+0000","last_peered":"2024-09-10T20:24:51.337928+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:51.337928+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7174,"log_dups_size":0,"ondisk_log_size":7174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40455920,"num_objects":4939,"num_object_clones":0,"num_object_copies":19756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7171,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7225","reported_seq":7416,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.963950+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:50.963950+0000","last_peered":"2024-09-10T20:24:50.963950+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:50.963950+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40467024,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7224,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'7053","reported_seq":7199,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.237437+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:53.237437+0000","last_peered":"2024-09-10T20:24:53.237437+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:53.237437+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39139920,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7052,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7185","reported_seq":7337,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.258204+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:51.258204+0000","last_peered":"2024-09-10T20:24:51.258204+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:51.258204+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7185,"log_dups_size":0,"ondisk_log_size":7185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7363","reported_seq":7509,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.119125+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:53.119125+0000","last_peered":"2024-09-10T20:24:53.119125+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:53.119125+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41661600,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3151}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7152","reported_seq":7271,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.159471+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:50.159471+0000","last_peered":"2024-09-10T20:24:50.159471+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:50.159471+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735904,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":650258600,"num_objects":79382,"num_object_clones":0,"num_object_copies":317528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102325,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1257099264,"data_stored":1257099264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115091,"ondisk_log_size":115091,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117061,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724004,"kb_used_data":318840,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93647836,"statfs":{"total":96636764160,"available":95895384064,"internally_reserved":0,"allocated":326492160,"data_stored":326288480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,5,5,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708256,"kb_used_data":316108,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93663584,"statfs":{"total":96636764160,"available":95911510016,"internally_reserved":0,"allocated":323694592,"data_stored":323490912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556852,"kb_used_data":276644,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93814988,"statfs":{"total":96636764160,"available":96066547712,"internally_reserved":0,"allocated":283283456,"data_stored":283079776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707828,"kb_used_data":319396,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93664012,"statfs":{"total":96636764160,"available":95911948288,"internally_reserved":0,"allocated":327061504,"data_stored":326857824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326049792,"data_stored":326049792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282832896,"data_stored":282832896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323092480,"data_stored":323092480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325124096,"data_stored":325124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:55.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:24:55.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'7167", 'reported_seq': 7332, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:50.992797+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:50.992797+0000', 'last_peered': '2024-09-10T20:24:50.992797+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:50.992797+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7167, 'log_dups_size': 0, 'ondisk_log_size': 7167, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40591360, 'num_objects': 4955, 'num_object_clones': 0, 'num_object_copies': 19820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1911, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7167, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3044}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:55.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:24:55.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'7221", 'reported_seq': 7372, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:53.297673+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:53.297673+0000', 'last_peered': '2024-09-10T20:24:53.297673+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:53.297673+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7221, 'log_dups_size': 0, 'ondisk_log_size': 7221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40295016, 'num_objects': 4920, 'num_object_clones': 0, 'num_object_copies': 19680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1932, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4920, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7220, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2989}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:55.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:24:55.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'7190", 'reported_seq': 7387, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:51.280915+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:51.280915+0000', 'last_peered': '2024-09-10T20:24:51.280915+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:51.280915+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7190, 'log_dups_size': 0, 'ondisk_log_size': 7190, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40785056, 'num_objects': 4979, 'num_object_clones': 0, 'num_object_copies': 19916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7188, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3065}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'7147", 'reported_seq': 7292, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:51.295259+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:51.295259+0000', 'last_peered': '2024-09-10T20:24:51.295259+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:51.295259+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7147, 'log_dups_size': 0, 'ondisk_log_size': 7147, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40558592, 'num_objects': 4951, 'num_object_clones': 0, 'num_object_copies': 19804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1893, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7147, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3058}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'7128", 'reported_seq': 7260, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:50.420125+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:50.420125+0000', 'last_peered': '2024-09-10T20:24:50.420125+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:50.420125+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7128, 'log_dups_size': 0, 'ondisk_log_size': 7128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40481952, 'num_objects': 4942, 'num_object_clones': 0, 'num_object_copies': 19768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1928, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7126, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3016}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'7289", 'reported_seq': 7404, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:50.285671+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:50.285671+0000', 'last_peered': '2024-09-10T20:24:50.285671+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:50.285671+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7289, 'log_dups_size': 0, 'ondisk_log_size': 7289, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40989856, 'num_objects': 5004, 'num_object_clones': 0, 'num_object_copies': 20016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2026, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7287, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'7209", 'reported_seq': 10804, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:53.286656+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:53.286656+0000', 'last_peered': '2024-09-10T20:24:53.286656+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:53.286656+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7209, 'log_dups_size': 0, 'ondisk_log_size': 7209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40424608, 'num_objects': 4935, 'num_object_clones': 0, 'num_object_copies': 19740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 153, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7207, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1754, 'num_bytes_recovered': 14336000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4782}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:24:55.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'7174", 'reported_seq': 7348, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:51.337928+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:51.337928+0000', 'last_peered': '2024-09-10T20:24:51.337928+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:51.337928+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7174, 'log_dups_size': 0, 'ondisk_log_size': 7174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40455920, 'num_objects': 4939, 'num_object_clones': 0, 'num_object_copies': 19756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1906, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7171, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3036}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'7225", 'reported_seq': 7416, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:50.963950+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:50.963950+0000', 'last_peered': '2024-09-10T20:24:50.963950+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:50.963950+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7225, 'log_dups_size': 0, 'ondisk_log_size': 7225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40467024, 'num_objects': 4940, 'num_object_clones': 0, 'num_object_copies': 19760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1912, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7224, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'7053", 'reported_seq': 7199, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:53.237437+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:53.237437+0000', 'last_peered': '2024-09-10T20:24:53.237437+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:53.237437+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7053, 'log_dups_size': 0, 'ondisk_log_size': 7053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39139920, 'num_objects': 4778, 'num_object_clones': 0, 'num_object_copies': 19112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1920, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7052, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'7185", 'reported_seq': 7337, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:51.258204+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:51.258204+0000', 'last_peered': '2024-09-10T20:24:51.258204+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:51.258204+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7185, 'log_dups_size': 0, 'ondisk_log_size': 7185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40211616, 'num_objects': 4909, 'num_object_clones': 0, 'num_object_copies': 19636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1933, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7183, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'7363", 'reported_seq': 7509, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:53.119125+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:53.119125+0000', 'last_peered': '2024-09-10T20:24:53.119125+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:53.119125+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7363, 'log_dups_size': 0, 'ondisk_log_size': 7363, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41661600, 'num_objects': 5086, 'num_object_clones': 0, 'num_object_copies': 20344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7361, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3151}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:24:55.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'7152", 'reported_seq': 7271, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:50.159471+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:50.159471+0000', 'last_peered': '2024-09-10T20:24:50.159471+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:50.159471+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7152, 'log_dups_size': 0, 'ondisk_log_size': 7152, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40735904, 'num_objects': 4973, 'num_object_clones': 0, 'num_object_copies': 19892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7150, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3015}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:24:55.158 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:24:55.881 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23327/317528 objects degraded (7.346%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":384,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79382,"data_bytes":650258600,"bytes_used":2761666560,"bytes_avail":383785390080,"bytes_total":386547056640,"degraded_objects":23327,"degraded_total":317528,"degraded_ratio":0.073464387392607897,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":174016,"recovering_keys_per_sec":0,"num_objects_recovered":326,"num_bytes_recovered":2088960,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-10T20:24:07.970414+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:24:55.909 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:24:55.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:24:56.503 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:24:56.504 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:24:56.526 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-10T20:24:55.988168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646899880,"num_objects":78972,"num_object_clones":0,"num_object_copies":315888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102735,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115501,"ondisk_log_size":115501,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717376,"kb_used_data":1225056,"kb_used_omap":14,"kb_used_meta":1492209,"kb_avail":374769984,"statfs":{"total":386547056640,"available":383764463616,"internally_reserved":0,"allocated":1254457344,"data_stored":1253642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1528022582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,6,6,14,7,0,0,0,0,4,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24322048,"num_objects":-2969,"num_object_clones":0,"num_object_copies":-11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004454"},"pg_stats":[{"pgid":"1.f","version":"38'7167","reported_seq":7332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.992797+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:50.992797+0000","last_peered":"2024-09-10T20:24:50.992797+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:50.992797+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7167,"log_dups_size":0,"ondisk_log_size":7167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40591360,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7294","reported_seq":7421,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:54.217537+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:54.217537+0000","last_peered":"2024-09-10T20:24:54.217537+0000","last_clean":"2024-09-10T20:24:54.217537+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:54.217537+0000","last_undegraded":"2024-09-10T20:24:54.217537+0000","last_fullsized":"2024-09-10T20:24:54.217537+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7294,"log_dups_size":0,"ondisk_log_size":7294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40615936,"num_objects":4958,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7221","reported_seq":7372,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.297673+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:53.297673+0000","last_peered":"2024-09-10T20:24:53.297673+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:53.297673+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40295016,"num_objects":4920,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7220,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2989}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7345","reported_seq":11509,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:54.123000+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:54.123000+0000","last_peered":"2024-09-10T20:24:54.123000+0000","last_clean":"2024-09-10T20:24:54.123000+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:54.123000+0000","last_undegraded":"2024-09-10T20:24:54.123000+0000","last_fullsized":"2024-09-10T20:24:54.123000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41459712,"num_objects":5061,"num_object_clones":0,"num_object_copies":20244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1605,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7190","reported_seq":7387,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.280915+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:51.280915+0000","last_peered":"2024-09-10T20:24:51.280915+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:51.280915+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7190,"log_dups_size":0,"ondisk_log_size":7190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7147","reported_seq":7292,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.295259+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:51.295259+0000","last_peered":"2024-09-10T20:24:51.295259+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:51.295259+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7147,"log_dups_size":0,"ondisk_log_size":7147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40558592,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7200","reported_seq":7334,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.410101+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:55.410101+0000","last_peered":"2024-09-10T20:24:55.410101+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:55.410101+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39892128,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7380","reported_seq":7495,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.432053+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:55.432053+0000","last_peered":"2024-09-10T20:24:55.432053+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:55.432053+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7380,"log_dups_size":0,"ondisk_log_size":7380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244384,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7114","reported_seq":11081,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:51.154556+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:51.154556+0000","last_peered":"2024-09-10T20:24:51.154556+0000","last_clean":"2024-09-10T20:24:51.154556+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:51.154556+0000","last_undegraded":"2024-09-10T20:24:51.154556+0000","last_fullsized":"2024-09-10T20:24:51.154556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40032848,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3667,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7209","reported_seq":10804,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.286656+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:53.286656+0000","last_peered":"2024-09-10T20:24:53.286656+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:53.286656+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7209,"log_dups_size":0,"ondisk_log_size":7209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40424608,"num_objects":4935,"num_object_clones":0,"num_object_copies":19740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7174","reported_seq":7348,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.337928+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:51.337928+0000","last_peered":"2024-09-10T20:24:51.337928+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:51.337928+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7174,"log_dups_size":0,"ondisk_log_size":7174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40455920,"num_objects":4939,"num_object_clones":0,"num_object_copies":19756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7171,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7225","reported_seq":7416,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:50.963950+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:50.963950+0000","last_peered":"2024-09-10T20:24:50.963950+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:50.963950+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40467024,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7224,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'7053","reported_seq":7199,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.237437+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:53.237437+0000","last_peered":"2024-09-10T20:24:53.237437+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:53.237437+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39139920,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7052,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7185","reported_seq":7337,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:51.258204+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:51.258204+0000","last_peered":"2024-09-10T20:24:51.258204+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:51.258204+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7185,"log_dups_size":0,"ondisk_log_size":7185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211616,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7363","reported_seq":7509,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:53.119125+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:53.119125+0000","last_peered":"2024-09-10T20:24:53.119125+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:53.119125+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41661600,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3151}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7234","reported_seq":7355,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.312969+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:55.312969+0000","last_peered":"2024-09-10T20:24:55.312969+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:55.312969+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7234,"log_dups_size":0,"ondisk_log_size":7234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40064160,"num_objects":4891,"num_object_clones":0,"num_object_copies":19564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":646899880,"num_objects":78972,"num_object_clones":0,"num_object_copies":315888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102735,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":14336000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1251061760,"data_stored":1251061760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115501,"ondisk_log_size":115501,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117061,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724004,"kb_used_data":318840,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93647836,"statfs":{"total":96636764160,"available":95895384064,"internally_reserved":0,"allocated":326492160,"data_stored":326288480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,5,5,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708256,"kb_used_data":316108,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93663584,"statfs":{"total":96636764160,"available":95911510016,"internally_reserved":0,"allocated":323694592,"data_stored":323490912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578208,"kb_used_data":275792,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93793632,"statfs":{"total":96636764160,"available":96044679168,"internally_reserved":0,"allocated":282411008,"data_stored":282207328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706908,"kb_used_data":314316,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93664932,"statfs":{"total":96636764160,"available":95912890368,"internally_reserved":0,"allocated":321859584,"data_stored":321655904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320983040,"data_stored":320983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281862144,"data_stored":281862144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323092480,"data_stored":323092480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325124096,"data_stored":325124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:24:59.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:00.137 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:00.167 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-10T20:24:59.989638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":638822568,"num_objects":77986,"num_object_clones":0,"num_object_copies":311944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103721,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2420,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116487,"ondisk_log_size":116487,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2706656,"kb_used_data":1214360,"kb_used_omap":14,"kb_used_meta":1492209,"kb_avail":374780704,"statfs":{"total":386547056640,"available":383775440896,"internally_reserved":0,"allocated":1243504640,"data_stored":1242689920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1528022582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,7,8,12,5,1,3,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24453120,"num_objects":-2985,"num_object_clones":0,"num_object_copies":-11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2985,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":359,"num_bytes_recovered":2277376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004173"},"pg_stats":[{"pgid":"1.f","version":"38'7251","reported_seq":7426,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.252816+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:56.252816+0000","last_peered":"2024-09-10T20:24:56.252816+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:56.252816+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7251,"log_dups_size":0,"ondisk_log_size":7251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7251,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7367","reported_seq":7494,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:58.922127+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:58.922127+0000","last_peered":"2024-09-10T20:24:58.922127+0000","last_clean":"2024-09-10T20:24:58.922127+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:58.922127+0000","last_undegraded":"2024-09-10T20:24:58.922127+0000","last_fullsized":"2024-09-10T20:24:58.922127+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40017920,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7299","reported_seq":7453,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.209496+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:58.209496+0000","last_peered":"2024-09-10T20:24:58.209496+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:58.209496+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39656040,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7298,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7436","reported_seq":11600,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:59.157098+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:59.157098+0000","last_peered":"2024-09-10T20:24:59.157098+0000","last_clean":"2024-09-10T20:24:59.157098+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:59.157098+0000","last_undegraded":"2024-09-10T20:24:59.157098+0000","last_fullsized":"2024-09-10T20:24:59.157098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40714240,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1696,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7240","reported_seq":7443,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.026347+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:56.026347+0000","last_peered":"2024-09-10T20:24:56.026347+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:56.026347+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40375456,"num_objects":4929,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7229","reported_seq":7380,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.369731+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:56.369731+0000","last_peered":"2024-09-10T20:24:56.369731+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:56.369731+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7229,"log_dups_size":0,"ondisk_log_size":7229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39886848,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7200","reported_seq":7334,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.410101+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:55.410101+0000","last_peered":"2024-09-10T20:24:55.410101+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:55.410101+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39892128,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7380","reported_seq":7495,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.432053+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:55.432053+0000","last_peered":"2024-09-10T20:24:55.432053+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:55.432053+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7380,"log_dups_size":0,"ondisk_log_size":7380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244384,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7192","reported_seq":11158,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:56.312192+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:56.312192+0000","last_peered":"2024-09-10T20:24:56.312192+0000","last_clean":"2024-09-10T20:24:56.312192+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:56.312192+0000","last_undegraded":"2024-09-10T20:24:56.312192+0000","last_fullsized":"2024-09-10T20:24:56.312192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39393872,"num_objects":4809,"num_object_clones":0,"num_object_copies":19236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3745,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7274","reported_seq":11100,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.299532+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:58.299532+0000","last_peered":"2024-09-10T20:24:58.299532+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:58.299532+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7274,"log_dups_size":0,"ondisk_log_size":7274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39892128,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7242","reported_seq":7438,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.131711+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:56.131711+0000","last_peered":"2024-09-10T20:24:56.131711+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:56.131711+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39898864,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7284","reported_seq":7491,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.033023+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:56.033023+0000","last_peered":"2024-09-10T20:24:56.033023+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:56.033023+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39983696,"num_objects":4881,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'7135","reported_seq":7285,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.128722+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:58.128722+0000","last_peered":"2024-09-10T20:24:58.128722+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:58.128722+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38468176,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7283","reported_seq":7455,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.325593+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:56.325593+0000","last_peered":"2024-09-10T20:24:56.325593+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:56.325593+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408800,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7281,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7441","reported_seq":7591,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.241588+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:58.241588+0000","last_peered":"2024-09-10T20:24:58.241588+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:58.241588+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41022624,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7234","reported_seq":7355,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.312969+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:55.312969+0000","last_peered":"2024-09-10T20:24:55.312969+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:55.312969+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7234,"log_dups_size":0,"ondisk_log_size":7234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40064160,"num_objects":4891,"num_object_clones":0,"num_object_copies":19564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":638822568,"num_objects":77986,"num_object_clones":0,"num_object_copies":311944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103721,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2420,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1240182784,"data_stored":1240182784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116487,"ondisk_log_size":116487,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706508,"kb_used_data":313844,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93665332,"statfs":{"total":96636764160,"available":95913299968,"internally_reserved":0,"allocated":321376256,"data_stored":321172576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,0,5,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707524,"kb_used_data":311220,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93664316,"statfs":{"total":96636764160,"available":95912259584,"internally_reserved":0,"allocated":318689280,"data_stored":318485600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585716,"kb_used_data":274980,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93786124,"statfs":{"total":96636764160,"available":96036990976,"internally_reserved":0,"allocated":281579520,"data_stored":281375840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706908,"kb_used_data":314316,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93664932,"statfs":{"total":96636764160,"available":95912890368,"internally_reserved":0,"allocated":321859584,"data_stored":321655904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320983040,"data_stored":320983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281104384,"data_stored":281104384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317972480,"data_stored":317972480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320122880,"data_stored":320122880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:00.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:00.757 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:00.758 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:00.778 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-10T20:24:59.989638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":638822568,"num_objects":77986,"num_object_clones":0,"num_object_copies":311944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103721,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2420,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116487,"ondisk_log_size":116487,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2706656,"kb_used_data":1214360,"kb_used_omap":14,"kb_used_meta":1492209,"kb_avail":374780704,"statfs":{"total":386547056640,"available":383775440896,"internally_reserved":0,"allocated":1243504640,"data_stored":1242689920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1528022582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,7,8,12,5,1,3,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24453120,"num_objects":-2985,"num_object_clones":0,"num_object_copies":-11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2985,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":359,"num_bytes_recovered":2277376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004173"},"pg_stats":[{"pgid":"1.f","version":"38'7251","reported_seq":7426,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.252816+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:24:56.252816+0000","last_peered":"2024-09-10T20:24:56.252816+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:24:56.252816+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7251,"log_dups_size":0,"ondisk_log_size":7251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7251,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7367","reported_seq":7494,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:58.922127+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:58.922127+0000","last_peered":"2024-09-10T20:24:58.922127+0000","last_clean":"2024-09-10T20:24:58.922127+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:58.922127+0000","last_undegraded":"2024-09-10T20:24:58.922127+0000","last_fullsized":"2024-09-10T20:24:58.922127+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40017920,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7299","reported_seq":7453,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.209496+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:58.209496+0000","last_peered":"2024-09-10T20:24:58.209496+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:58.209496+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39656040,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7298,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7436","reported_seq":11600,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:59.157098+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:59.157098+0000","last_peered":"2024-09-10T20:24:59.157098+0000","last_clean":"2024-09-10T20:24:59.157098+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:59.157098+0000","last_undegraded":"2024-09-10T20:24:59.157098+0000","last_fullsized":"2024-09-10T20:24:59.157098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40714240,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1696,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7240","reported_seq":7443,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.026347+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:24:56.026347+0000","last_peered":"2024-09-10T20:24:56.026347+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:24:56.026347+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40375456,"num_objects":4929,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7229","reported_seq":7380,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.369731+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:24:56.369731+0000","last_peered":"2024-09-10T20:24:56.369731+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:24:56.369731+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7229,"log_dups_size":0,"ondisk_log_size":7229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39886848,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7200","reported_seq":7334,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.410101+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:24:55.410101+0000","last_peered":"2024-09-10T20:24:55.410101+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:24:55.410101+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39892128,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7380","reported_seq":7495,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.432053+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:24:55.432053+0000","last_peered":"2024-09-10T20:24:55.432053+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:24:55.432053+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7380,"log_dups_size":0,"ondisk_log_size":7380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244384,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7192","reported_seq":11158,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:56.312192+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:24:56.312192+0000","last_peered":"2024-09-10T20:24:56.312192+0000","last_clean":"2024-09-10T20:24:56.312192+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:24:56.312192+0000","last_undegraded":"2024-09-10T20:24:56.312192+0000","last_fullsized":"2024-09-10T20:24:56.312192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39393872,"num_objects":4809,"num_object_clones":0,"num_object_copies":19236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3745,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7274","reported_seq":11100,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.299532+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:58.299532+0000","last_peered":"2024-09-10T20:24:58.299532+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:58.299532+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7274,"log_dups_size":0,"ondisk_log_size":7274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39892128,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7242","reported_seq":7438,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.131711+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:24:56.131711+0000","last_peered":"2024-09-10T20:24:56.131711+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:24:56.131711+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39898864,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7284","reported_seq":7491,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.033023+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:24:56.033023+0000","last_peered":"2024-09-10T20:24:56.033023+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:24:56.033023+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39983696,"num_objects":4881,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'7135","reported_seq":7285,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.128722+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:58.128722+0000","last_peered":"2024-09-10T20:24:58.128722+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:58.128722+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38468176,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7283","reported_seq":7455,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:56.325593+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:24:56.325593+0000","last_peered":"2024-09-10T20:24:56.325593+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:24:56.325593+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408800,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7281,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7441","reported_seq":7591,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.241588+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:58.241588+0000","last_peered":"2024-09-10T20:24:58.241588+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:58.241588+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41022624,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7234","reported_seq":7355,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:55.312969+0000","last_change":"2024-09-10T20:20:16.154800+0000","last_active":"2024-09-10T20:24:55.312969+0000","last_peered":"2024-09-10T20:24:55.312969+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:24:55.312969+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7234,"log_dups_size":0,"ondisk_log_size":7234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40064160,"num_objects":4891,"num_object_clones":0,"num_object_copies":19564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":638822568,"num_objects":77986,"num_object_clones":0,"num_object_copies":311944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103721,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2420,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1240182784,"data_stored":1240182784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116487,"ondisk_log_size":116487,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706508,"kb_used_data":313844,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93665332,"statfs":{"total":96636764160,"available":95913299968,"internally_reserved":0,"allocated":321376256,"data_stored":321172576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,0,5,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707524,"kb_used_data":311220,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93664316,"statfs":{"total":96636764160,"available":95912259584,"internally_reserved":0,"allocated":318689280,"data_stored":318485600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585716,"kb_used_data":274980,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93786124,"statfs":{"total":96636764160,"available":96036990976,"internally_reserved":0,"allocated":281579520,"data_stored":281375840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706908,"kb_used_data":314316,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93664932,"statfs":{"total":96636764160,"available":95912890368,"internally_reserved":0,"allocated":321859584,"data_stored":321655904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320983040,"data_stored":320983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281104384,"data_stored":281104384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317972480,"data_stored":317972480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320122880,"data_stored":320122880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:00.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:00.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'7251", 'reported_seq': 7426, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:56.252816+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:24:56.252816+0000', 'last_peered': '2024-09-10T20:24:56.252816+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:24:56.252816+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7251, 'log_dups_size': 0, 'ondisk_log_size': 7251, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39903232, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1906, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7251, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2965}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'7299", 'reported_seq': 7453, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:58.209496+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:24:58.209496+0000', 'last_peered': '2024-09-10T20:24:58.209496+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:24:58.209496+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7299, 'log_dups_size': 0, 'ondisk_log_size': 7299, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39656040, 'num_objects': 4842, 'num_object_clones': 0, 'num_object_copies': 19368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1930, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4842, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7298, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2913}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'7240", 'reported_seq': 7443, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:56.026347+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:24:56.026347+0000', 'last_peered': '2024-09-10T20:24:56.026347+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:24:56.026347+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7240, 'log_dups_size': 0, 'ondisk_log_size': 7240, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40375456, 'num_objects': 4929, 'num_object_clones': 0, 'num_object_copies': 19716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1913, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7238, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3018}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'7229", 'reported_seq': 7380, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:56.369731+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:24:56.369731+0000', 'last_peered': '2024-09-10T20:24:56.369731+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:24:56.369731+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7229, 'log_dups_size': 0, 'ondisk_log_size': 7229, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39886848, 'num_objects': 4869, 'num_object_clones': 0, 'num_object_copies': 19476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7229, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2979}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'7200", 'reported_seq': 7334, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:55.410101+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:24:55.410101+0000', 'last_peered': '2024-09-10T20:24:55.410101+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:24:55.410101+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7200, 'log_dups_size': 0, 'ondisk_log_size': 7200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39892128, 'num_objects': 4870, 'num_object_clones': 0, 'num_object_copies': 19480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2945}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:00.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'7380", 'reported_seq': 7495, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:55.432053+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:24:55.432053+0000', 'last_peered': '2024-09-10T20:24:55.432053+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:24:55.432053+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7380, 'log_dups_size': 0, 'ondisk_log_size': 7380, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40244384, 'num_objects': 4913, 'num_object_clones': 0, 'num_object_copies': 19652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2026, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7378, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'7274", 'reported_seq': 11100, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:58.299532+0000', 'last_change': '2024-09-10T20:23:11.162827+0000', 'last_active': '2024-09-10T20:24:58.299532+0000', 'last_peered': '2024-09-10T20:24:58.299532+0000', 'last_clean': '2024-09-10T20:18:57.966015+0000', 'last_became_active': '2024-09-10T20:20:16.098986+0000', 'last_became_peered': '2024-09-10T20:20:16.098986+0000', 'last_unstale': '2024-09-10T20:24:58.299532+0000', 'last_undegraded': '2024-09-10T20:20:15.485998+0000', 'last_fullsized': '2024-09-10T20:20:15.464227+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7274, 'log_dups_size': 0, 'ondisk_log_size': 7274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39892128, 'num_objects': 4870, 'num_object_clones': 0, 'num_object_copies': 19480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 37, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7272, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1870, 'num_bytes_recovered': 15286272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4833}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'7242", 'reported_seq': 7438, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:56.131711+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:24:56.131711+0000', 'last_peered': '2024-09-10T20:24:56.131711+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:24:56.131711+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7242, 'log_dups_size': 0, 'ondisk_log_size': 7242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39898864, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7239, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2979}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'7284", 'reported_seq': 7491, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:56.033023+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:24:56.033023+0000', 'last_peered': '2024-09-10T20:24:56.033023+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:24:56.033023+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7284, 'log_dups_size': 0, 'ondisk_log_size': 7284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39983696, 'num_objects': 4881, 'num_object_clones': 0, 'num_object_copies': 19524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7283, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'7135", 'reported_seq': 7285, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:58.128722+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:24:58.128722+0000', 'last_peered': '2024-09-10T20:24:58.128722+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:24:58.128722+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7135, 'log_dups_size': 0, 'ondisk_log_size': 7135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38468176, 'num_objects': 4696, 'num_object_clones': 0, 'num_object_copies': 18784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1918, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7134, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'7283", 'reported_seq': 7455, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:56.325593+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:24:56.325593+0000', 'last_peered': '2024-09-10T20:24:56.325593+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:24:56.325593+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7283, 'log_dups_size': 0, 'ondisk_log_size': 7283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39408800, 'num_objects': 4811, 'num_object_clones': 0, 'num_object_copies': 19244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1923, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7281, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:00.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'7441", 'reported_seq': 7591, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:58.241588+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:24:58.241588+0000', 'last_peered': '2024-09-10T20:24:58.241588+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:24:58.241588+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7441, 'log_dups_size': 0, 'ondisk_log_size': 7441, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41022624, 'num_objects': 5008, 'num_object_clones': 0, 'num_object_copies': 20032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7439, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:00.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:00.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'7234", 'reported_seq': 7355, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:24:55.312969+0000', 'last_change': '2024-09-10T20:20:16.154800+0000', 'last_active': '2024-09-10T20:24:55.312969+0000', 'last_peered': '2024-09-10T20:24:55.312969+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:24:55.312969+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7234, 'log_dups_size': 0, 'ondisk_log_size': 7234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40064160, 'num_objects': 4891, 'num_object_clones': 0, 'num_object_copies': 19564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7232, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:00.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:01.531 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:01.531 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23163/311944 objects degraded (7.425%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":390,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77986,"data_bytes":638822568,"bytes_used":2771615744,"bytes_avail":383775440896,"bytes_total":386547056640,"degraded_objects":23163,"degraded_total":311944,"degraded_ratio":0.074253712204754699,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":189715,"recovering_keys_per_sec":0,"num_objects_recovered":359,"num_bytes_recovered":2277376,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":248},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:01.556 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:01.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:02.139 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:02.140 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:02.165 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-10T20:25:01.991267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632293544,"num_objects":77189,"num_object_clones":0,"num_object_copies":308756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104518,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117284,"ondisk_log_size":117284,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704948,"kb_used_data":1204328,"kb_used_omap":14,"kb_used_meta":1500529,"kb_avail":374782412,"statfs":{"total":386547056640,"available":383777189888,"internally_reserved":0,"allocated":1233231872,"data_stored":1232417152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1536542262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,7,1,11,9,13,2,3,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20652032,"num_objects":-2521,"num_object_clones":0,"num_object_copies":-10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004271"},"pg_stats":[{"pgid":"1.f","version":"38'7338","reported_seq":7523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.329585+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:01.329585+0000","last_peered":"2024-09-10T20:25:01.329585+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:01.329585+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7367","reported_seq":7494,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:58.922127+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:58.922127+0000","last_peered":"2024-09-10T20:24:58.922127+0000","last_clean":"2024-09-10T20:24:58.922127+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:58.922127+0000","last_undegraded":"2024-09-10T20:24:58.922127+0000","last_fullsized":"2024-09-10T20:24:58.922127+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40017920,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'7299","reported_seq":7453,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.209496+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:24:58.209496+0000","last_peered":"2024-09-10T20:24:58.209496+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:24:58.209496+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39656040,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7298,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7436","reported_seq":11600,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:59.157098+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:59.157098+0000","last_peered":"2024-09-10T20:24:59.157098+0000","last_clean":"2024-09-10T20:24:59.157098+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:59.157098+0000","last_undegraded":"2024-09-10T20:24:59.157098+0000","last_fullsized":"2024-09-10T20:24:59.157098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40714240,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1696,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7330","reported_seq":7543,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.351482+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:01.351482+0000","last_peered":"2024-09-10T20:25:01.351482+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:01.351482+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39638176,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7296","reported_seq":7457,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.327742+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:01.327742+0000","last_peered":"2024-09-10T20:25:01.327742+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:01.327742+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39337984,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7282","reported_seq":7422,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:00.393709+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:00.393709+0000","last_peered":"2024-09-10T20:25:00.393709+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:00.393709+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39220384,"num_objects":4788,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7280,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7457","reported_seq":7578,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:00.469910+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:00.469910+0000","last_peered":"2024-09-10T20:25:00.469910+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:00.469910+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7457,"log_dups_size":0,"ondisk_log_size":7457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39613600,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7455,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7264","reported_seq":11233,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T20:25:01.325050+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:01.325050+0000","last_peered":"2024-09-10T20:25:01.325050+0000","last_clean":"2024-09-10T20:25:01.325050+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:01.325050+0000","last_undegraded":"2024-09-10T20:25:01.325050+0000","last_fullsized":"2024-09-10T20:25:01.325050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38804048,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3817,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7274","reported_seq":11100,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.299532+0000","last_change":"2024-09-10T20:23:11.162827+0000","last_active":"2024-09-10T20:24:58.299532+0000","last_peered":"2024-09-10T20:24:58.299532+0000","last_clean":"2024-09-10T20:18:57.966015+0000","last_became_active":"2024-09-10T20:20:16.098986+0000","last_became_peered":"2024-09-10T20:20:16.098986+0000","last_unstale":"2024-09-10T20:24:58.299532+0000","last_undegraded":"2024-09-10T20:20:15.485998+0000","last_fullsized":"2024-09-10T20:20:15.464227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7274,"log_dups_size":0,"ondisk_log_size":7274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39892128,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7306","reported_seq":7510,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339326+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:01.339326+0000","last_peered":"2024-09-10T20:25:01.339326+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:01.339326+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39374576,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7303,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7367","reported_seq":7582,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339630+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:01.339630+0000","last_peered":"2024-09-10T20:25:01.339630+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:01.339630+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303760,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'7135","reported_seq":7285,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.128722+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:24:58.128722+0000","last_peered":"2024-09-10T20:24:58.128722+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:24:58.128722+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38468176,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7357","reported_seq":7545,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.350427+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:01.350427+0000","last_peered":"2024-09-10T20:25:01.350427+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:01.350427+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7357,"log_dups_size":0,"ondisk_log_size":7357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38802592,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7355,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7441","reported_seq":7591,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:24:58.241588+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:24:58.241588+0000","last_peered":"2024-09-10T20:24:58.241588+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:24:58.241588+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41022624,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7335","reported_seq":7458,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:00.468366+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:00.468366+0000","last_peered":"2024-09-10T20:25:00.468366+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:00.468366+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7335,"log_dups_size":0,"ondisk_log_size":7335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39236768,"num_objects":4790,"num_object_clones":0,"num_object_copies":19160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7333,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":632293544,"num_objects":77189,"num_object_clones":0,"num_object_copies":308756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104518,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":15286272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1229713408,"data_stored":1229713408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117284,"ondisk_log_size":117284,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705624,"kb_used_data":308800,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93666216,"statfs":{"total":96636764160,"available":95914205184,"internally_reserved":0,"allocated":316211200,"data_stored":316007520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,7,0,1,1,2,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707524,"kb_used_data":311220,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93664316,"statfs":{"total":96636764160,"available":95912259584,"internally_reserved":0,"allocated":318689280,"data_stored":318485600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585716,"kb_used_data":274980,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93786124,"statfs":{"total":96636764160,"available":96036990976,"internally_reserved":0,"allocated":281579520,"data_stored":281375840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706084,"kb_used_data":309328,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93665756,"statfs":{"total":96636764160,"available":95913734144,"internally_reserved":0,"allocated":316751872,"data_stored":316548192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,1,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315699200,"data_stored":315699200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281104384,"data_stored":281104384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317972480,"data_stored":317972480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314937344,"data_stored":314937344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:05.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:05.709 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:05.710 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:05.737 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-10T20:25:03.991936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":629999784,"num_objects":76909,"num_object_clones":0,"num_object_copies":307636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104798,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2491,"num_bytes_recovered":15589376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117564,"ondisk_log_size":117564,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722248,"kb_used_data":1199380,"kb_used_omap":14,"kb_used_meta":1522737,"kb_avail":374765112,"statfs":{"total":386547056640,"available":383759474688,"internally_reserved":0,"allocated":1228165120,"data_stored":1227341019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1559283254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,7,0,14,7,9,2,3,0,4,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16900096,"num_objects":-2063,"num_object_clones":0,"num_object_copies":-8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2063,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003767"},"pg_stats":[{"pgid":"1.f","version":"38'7338","reported_seq":7523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.329585+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:01.329585+0000","last_peered":"2024-09-10T20:25:01.329585+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:01.329585+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7367","reported_seq":7494,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:58.922127+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:24:58.922127+0000","last_peered":"2024-09-10T20:24:58.922127+0000","last_clean":"2024-09-10T20:24:58.922127+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:24:58.922127+0000","last_undegraded":"2024-09-10T20:24:58.922127+0000","last_fullsized":"2024-09-10T20:24:58.922127+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40017920,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7385","reported_seq":7547,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.314151+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:03.314151+0000","last_peered":"2024-09-10T20:25:03.314151+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:03.314151+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951528,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7384,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2828}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7436","reported_seq":11600,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-10T20:24:59.157098+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:24:59.157098+0000","last_peered":"2024-09-10T20:24:59.157098+0000","last_clean":"2024-09-10T20:24:59.157098+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:24:59.157098+0000","last_undegraded":"2024-09-10T20:24:59.157098+0000","last_fullsized":"2024-09-10T20:24:59.157098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40714240,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1696,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7330","reported_seq":7543,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.351482+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:01.351482+0000","last_peered":"2024-09-10T20:25:01.351482+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:01.351482+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39638176,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7296","reported_seq":7457,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.327742+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:01.327742+0000","last_peered":"2024-09-10T20:25:01.327742+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:01.327742+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39337984,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7282","reported_seq":7422,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:00.393709+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:00.393709+0000","last_peered":"2024-09-10T20:25:00.393709+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:00.393709+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39220384,"num_objects":4788,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7280,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'7457","reported_seq":7578,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:00.469910+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:00.469910+0000","last_peered":"2024-09-10T20:25:00.469910+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:00.469910+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7457,"log_dups_size":0,"ondisk_log_size":7457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39613600,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7455,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7264","reported_seq":11233,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T20:25:01.325050+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:01.325050+0000","last_peered":"2024-09-10T20:25:01.325050+0000","last_clean":"2024-09-10T20:25:01.325050+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:01.325050+0000","last_undegraded":"2024-09-10T20:25:01.325050+0000","last_fullsized":"2024-09-10T20:25:01.325050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38804048,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3817,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7342","reported_seq":11262,"reported_epoch":40,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:25:03.347127+0000","last_change":"2024-09-10T20:25:03.051594+0000","last_active":"2024-09-10T20:25:03.347127+0000","last_peered":"2024-09-10T20:25:03.347127+0000","last_clean":"2024-09-10T20:25:03.347127+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:03.347127+0000","last_undegraded":"2024-09-10T20:25:03.347127+0000","last_fullsized":"2024-09-10T20:25:03.347127+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":299,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335072,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7340,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":15589376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7306","reported_seq":7510,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339326+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:01.339326+0000","last_peered":"2024-09-10T20:25:01.339326+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:01.339326+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39374576,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7303,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7367","reported_seq":7582,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339630+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:01.339630+0000","last_peered":"2024-09-10T20:25:01.339630+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:01.339630+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303760,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7203","reported_seq":7357,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.302341+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:03.302341+0000","last_peered":"2024-09-10T20:25:03.302341+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:03.302341+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37911120,"num_objects":4628,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7357","reported_seq":7545,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.350427+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:01.350427+0000","last_peered":"2024-09-10T20:25:01.350427+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:01.350427+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7357,"log_dups_size":0,"ondisk_log_size":7357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38802592,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7355,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7499","reported_seq":7653,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.266587+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:03.266587+0000","last_peered":"2024-09-10T20:25:03.266587+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:03.266587+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547488,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'7335","reported_seq":7458,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:00.468366+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:00.468366+0000","last_peered":"2024-09-10T20:25:00.468366+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:00.468366+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7335,"log_dups_size":0,"ondisk_log_size":7335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39236768,"num_objects":4790,"num_object_clones":0,"num_object_copies":19160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7333,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":629999784,"num_objects":76909,"num_object_clones":0,"num_object_copies":307636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104798,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2491,"num_bytes_recovered":15589376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1224691712,"data_stored":1224691712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117564,"ondisk_log_size":117564,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705624,"kb_used_data":308800,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93666216,"statfs":{"total":96636764160,"available":95914205184,"internally_reserved":0,"allocated":316211200,"data_stored":316007520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,7,0,1,1,2,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724824,"kb_used_data":306272,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93647016,"statfs":{"total":96636764160,"available":95894544384,"internally_reserved":0,"allocated":313622528,"data_stored":313409467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585716,"kb_used_data":274980,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93786124,"statfs":{"total":96636764160,"available":96036990976,"internally_reserved":0,"allocated":281579520,"data_stored":281375840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706084,"kb_used_data":309328,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93665756,"statfs":{"total":96636764160,"available":95913734144,"internally_reserved":0,"allocated":316751872,"data_stored":316548192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,1,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315699200,"data_stored":315699200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281104384,"data_stored":281104384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312950784,"data_stored":312950784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314937344,"data_stored":314937344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:05.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:06.325 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:06.326 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:06.357 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-10T20:25:05.992631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626722984,"num_objects":76509,"num_object_clones":0,"num_object_copies":306036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105198,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2592,"num_bytes_recovered":16343040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117964,"ondisk_log_size":117964,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726208,"kb_used_data":1193604,"kb_used_omap":14,"kb_used_meta":1532465,"kb_avail":374761152,"statfs":{"total":386547056640,"available":383755419648,"internally_reserved":0,"allocated":1222250496,"data_stored":1221407633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1569244726},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,8,1,6,5,10,4,3,0,3,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15925248,"num_objects":-1944,"num_object_clones":0,"num_object_copies":-7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":2007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003685"},"pg_stats":[{"pgid":"1.f","version":"38'7338","reported_seq":7523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.329585+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:01.329585+0000","last_peered":"2024-09-10T20:25:01.329585+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:01.329585+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7454","reported_seq":7585,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:04.185788+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:04.185788+0000","last_peered":"2024-09-10T20:25:04.185788+0000","last_clean":"2024-09-10T20:25:04.185788+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:04.185788+0000","last_undegraded":"2024-09-10T20:25:04.185788+0000","last_fullsized":"2024-09-10T20:25:04.185788+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39305216,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7385","reported_seq":7547,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.314151+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:03.314151+0000","last_peered":"2024-09-10T20:25:03.314151+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:03.314151+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951528,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7384,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2828}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7531","reported_seq":11699,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:04.142998+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:04.142998+0000","last_peered":"2024-09-10T20:25:04.142998+0000","last_clean":"2024-09-10T20:25:04.142998+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:04.142998+0000","last_undegraded":"2024-09-10T20:25:04.142998+0000","last_fullsized":"2024-09-10T20:25:04.142998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39936000,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1791,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7330","reported_seq":7543,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.351482+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:01.351482+0000","last_peered":"2024-09-10T20:25:01.351482+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:01.351482+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39638176,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7296","reported_seq":7457,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.327742+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:01.327742+0000","last_peered":"2024-09-10T20:25:01.327742+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:01.327742+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39337984,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7360","reported_seq":7508,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.475621+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:05.475621+0000","last_peered":"2024-09-10T20:25:05.475621+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:05.475621+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581408,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7358,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2790}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7527","reported_seq":7658,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.543179+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:05.543179+0000","last_peered":"2024-09-10T20:25:05.543179+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:05.543179+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7527,"log_dups_size":0,"ondisk_log_size":7527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39040160,"num_objects":4766,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7525,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7264","reported_seq":11233,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T20:25:01.325050+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:01.325050+0000","last_peered":"2024-09-10T20:25:01.325050+0000","last_clean":"2024-09-10T20:25:01.325050+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:01.325050+0000","last_undegraded":"2024-09-10T20:25:01.325050+0000","last_fullsized":"2024-09-10T20:25:01.325050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38804048,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3817,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7342","reported_seq":11262,"reported_epoch":40,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:25:03.347127+0000","last_change":"2024-09-10T20:25:03.051594+0000","last_active":"2024-09-10T20:25:03.347127+0000","last_peered":"2024-09-10T20:25:03.347127+0000","last_clean":"2024-09-10T20:25:03.347127+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:03.347127+0000","last_undegraded":"2024-09-10T20:25:03.347127+0000","last_fullsized":"2024-09-10T20:25:03.347127+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":299,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335072,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7340,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":15589376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7306","reported_seq":7510,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339326+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:01.339326+0000","last_peered":"2024-09-10T20:25:01.339326+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:01.339326+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39374576,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7303,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7367","reported_seq":7582,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339630+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:01.339630+0000","last_peered":"2024-09-10T20:25:01.339630+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:01.339630+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303760,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7203","reported_seq":7357,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.302341+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:03.302341+0000","last_peered":"2024-09-10T20:25:03.302341+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:03.302341+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37911120,"num_objects":4628,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7357","reported_seq":7545,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.350427+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:01.350427+0000","last_peered":"2024-09-10T20:25:01.350427+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:01.350427+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7357,"log_dups_size":0,"ondisk_log_size":7357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38802592,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7355,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7499","reported_seq":7653,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.266587+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:03.266587+0000","last_peered":"2024-09-10T20:25:03.266587+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:03.266587+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547488,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7405","reported_seq":7727,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.590504+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:05.590504+0000","last_peered":"2024-09-10T20:25:05.590504+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:05.590504+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663328,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7403,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":626722984,"num_objects":76509,"num_object_clones":0,"num_object_copies":306036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105198,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2592,"num_bytes_recovered":16343040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1218797568,"data_stored":1218797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117964,"ondisk_log_size":117964,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705624,"kb_used_data":308800,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93666216,"statfs":{"total":96636764160,"available":95914205184,"internally_reserved":0,"allocated":316211200,"data_stored":316007520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,7,0,1,1,2,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724824,"kb_used_data":306272,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93647016,"statfs":{"total":96636764160,"available":95894544384,"internally_reserved":0,"allocated":313622528,"data_stored":313409467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572188,"kb_used_data":273928,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93799652,"statfs":{"total":96636764160,"available":96050843648,"internally_reserved":0,"allocated":280502272,"data_stored":280289211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305393362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723572,"kb_used_data":304604,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93648268,"statfs":{"total":96636764160,"available":95895826432,"internally_reserved":0,"allocated":311914496,"data_stored":311701435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,4,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310988800,"data_stored":310988800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279920640,"data_stored":279920640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312950784,"data_stored":312950784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314937344,"data_stored":314937344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:06.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:06.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'7338", 'reported_seq': 7523, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:01.329585+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:01.329585+0000', 'last_peered': '2024-09-10T20:25:01.329585+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:01.329585+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7338, 'log_dups_size': 0, 'ondisk_log_size': 7338, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39190528, 'num_objects': 4784, 'num_object_clones': 0, 'num_object_copies': 19136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1902, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7338, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 64, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2882}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:06.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7385", 'reported_seq': 7547, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:03.314151+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:03.314151+0000', 'last_peered': '2024-09-10T20:25:03.314151+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:03.314151+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7385, 'log_dups_size': 0, 'ondisk_log_size': 7385, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38951528, 'num_objects': 4756, 'num_object_clones': 0, 'num_object_copies': 19024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1929, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4756, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7384, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2828}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'7330", 'reported_seq': 7543, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:01.351482+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:01.351482+0000', 'last_peered': '2024-09-10T20:25:01.351482+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:01.351482+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7330, 'log_dups_size': 0, 'ondisk_log_size': 7330, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39638176, 'num_objects': 4839, 'num_object_clones': 0, 'num_object_copies': 19356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7328, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 75, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2932}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'7296", 'reported_seq': 7457, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:01.327742+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:01.327742+0000', 'last_peered': '2024-09-10T20:25:01.327742+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:01.327742+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7296, 'log_dups_size': 0, 'ondisk_log_size': 7296, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39337984, 'num_objects': 4802, 'num_object_clones': 0, 'num_object_copies': 19208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1886, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7296, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2916}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7360", 'reported_seq': 7508, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:05.475621+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:05.475621+0000', 'last_peered': '2024-09-10T20:25:05.475621+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:05.475621+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7360, 'log_dups_size': 0, 'ondisk_log_size': 7360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38581408, 'num_objects': 4710, 'num_object_clones': 0, 'num_object_copies': 18840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1922, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7358, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2790}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:06.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'7527", 'reported_seq': 7658, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:05.543179+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:05.543179+0000', 'last_peered': '2024-09-10T20:25:05.543179+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:05.543179+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7527, 'log_dups_size': 0, 'ondisk_log_size': 7527, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39040160, 'num_objects': 4766, 'num_object_clones': 0, 'num_object_copies': 19064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7525, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'7342", 'reported_seq': 11262, 'reported_epoch': 40, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-10T20:25:03.347127+0000', 'last_change': '2024-09-10T20:25:03.051594+0000', 'last_active': '2024-09-10T20:25:03.347127+0000', 'last_peered': '2024-09-10T20:25:03.347127+0000', 'last_clean': '2024-09-10T20:25:03.347127+0000', 'last_became_active': '2024-09-10T20:25:02.385224+0000', 'last_became_peered': '2024-09-10T20:25:02.385224+0000', 'last_unstale': '2024-09-10T20:25:03.347127+0000', 'last_undegraded': '2024-09-10T20:25:03.347127+0000', 'last_fullsized': '2024-09-10T20:25:03.347127+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 40, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 299, 'log_size': 7342, 'log_dups_size': 0, 'ondisk_log_size': 7342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39335072, 'num_objects': 4802, 'num_object_clones': 0, 'num_object_copies': 19208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7340, 'num_write_kb': 48566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1907, 'num_bytes_recovered': 15589376, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'7306", 'reported_seq': 7510, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:01.339326+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:01.339326+0000', 'last_peered': '2024-09-10T20:25:01.339326+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:01.339326+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7306, 'log_dups_size': 0, 'ondisk_log_size': 7306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39374576, 'num_objects': 4807, 'num_object_clones': 0, 'num_object_copies': 19228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1892, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7303, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 70, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2918}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'7367", 'reported_seq': 7582, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:01.339630+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:01.339630+0000', 'last_peered': '2024-09-10T20:25:01.339630+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:01.339630+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7367, 'log_dups_size': 0, 'ondisk_log_size': 7367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39303760, 'num_objects': 4798, 'num_object_clones': 0, 'num_object_copies': 19192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1901, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7366, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7203", 'reported_seq': 7357, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:03.302341+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:03.302341+0000', 'last_peered': '2024-09-10T20:25:03.302341+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:03.302341+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7203, 'log_dups_size': 0, 'ondisk_log_size': 7203, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37911120, 'num_objects': 4628, 'num_object_clones': 0, 'num_object_copies': 18512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1918, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7202, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2711}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:06.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:06.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'7357", 'reported_seq': 7545, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:01.350427+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:01.350427+0000', 'last_peered': '2024-09-10T20:25:01.350427+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:01.350427+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7357, 'log_dups_size': 0, 'ondisk_log_size': 7357, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38802592, 'num_objects': 4737, 'num_object_clones': 0, 'num_object_copies': 18948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7355, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2823}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:06.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:06.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'7499", 'reported_seq': 7653, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:03.266587+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:03.266587+0000', 'last_peered': '2024-09-10T20:25:03.266587+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:03.266587+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7499, 'log_dups_size': 0, 'ondisk_log_size': 7499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40547488, 'num_objects': 4950, 'num_object_clones': 0, 'num_object_copies': 19800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7497, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:06.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:06.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7405", 'reported_seq': 7727, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:05.590504+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:05.590504+0000', 'last_peered': '2024-09-10T20:25:05.590504+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:05.590504+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7405, 'log_dups_size': 0, 'ondisk_log_size': 7405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38663328, 'num_objects': 4720, 'num_object_clones': 0, 'num_object_copies': 18880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7403, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 122, 'num_bytes_recovered': 753664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:06.361 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:07.098 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:07.098 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22991/306036 objects degraded (7.513%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":395,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":3},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76509,"data_bytes":626722984,"bytes_used":2791636992,"bytes_avail":383755419648,"bytes_total":386547056640,"degraded_objects":22991,"degraded_total":306036,"degraded_ratio":0.075125148675319248,"recovering_objects_per_sec":36,"recovering_bytes_per_sec":250764,"recovering_keys_per_sec":0,"num_objects_recovered":294,"num_bytes_recovered":2007040,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":242},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:07.125 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:07.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:07.706 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:07.706 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:07.731 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-10T20:25:05.992631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626722984,"num_objects":76509,"num_object_clones":0,"num_object_copies":306036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105198,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2592,"num_bytes_recovered":16343040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117964,"ondisk_log_size":117964,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726208,"kb_used_data":1193604,"kb_used_omap":14,"kb_used_meta":1532465,"kb_avail":374761152,"statfs":{"total":386547056640,"available":383755419648,"internally_reserved":0,"allocated":1222250496,"data_stored":1221407633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1569244726},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,8,1,6,5,10,4,3,0,3,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15925248,"num_objects":-1944,"num_object_clones":0,"num_object_copies":-7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":2007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003685"},"pg_stats":[{"pgid":"1.f","version":"38'7338","reported_seq":7523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.329585+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:01.329585+0000","last_peered":"2024-09-10T20:25:01.329585+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:01.329585+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7454","reported_seq":7585,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:04.185788+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:04.185788+0000","last_peered":"2024-09-10T20:25:04.185788+0000","last_clean":"2024-09-10T20:25:04.185788+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:04.185788+0000","last_undegraded":"2024-09-10T20:25:04.185788+0000","last_fullsized":"2024-09-10T20:25:04.185788+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39305216,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7385","reported_seq":7547,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.314151+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:03.314151+0000","last_peered":"2024-09-10T20:25:03.314151+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:03.314151+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951528,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7384,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2828}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7531","reported_seq":11699,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:04.142998+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:04.142998+0000","last_peered":"2024-09-10T20:25:04.142998+0000","last_clean":"2024-09-10T20:25:04.142998+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:04.142998+0000","last_undegraded":"2024-09-10T20:25:04.142998+0000","last_fullsized":"2024-09-10T20:25:04.142998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39936000,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1791,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7330","reported_seq":7543,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.351482+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:01.351482+0000","last_peered":"2024-09-10T20:25:01.351482+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:01.351482+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39638176,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7296","reported_seq":7457,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.327742+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:01.327742+0000","last_peered":"2024-09-10T20:25:01.327742+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:01.327742+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39337984,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7360","reported_seq":7508,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.475621+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:05.475621+0000","last_peered":"2024-09-10T20:25:05.475621+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:05.475621+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581408,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7358,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2790}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7527","reported_seq":7658,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.543179+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:05.543179+0000","last_peered":"2024-09-10T20:25:05.543179+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:05.543179+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7527,"log_dups_size":0,"ondisk_log_size":7527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39040160,"num_objects":4766,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7525,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7264","reported_seq":11233,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T20:25:01.325050+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:01.325050+0000","last_peered":"2024-09-10T20:25:01.325050+0000","last_clean":"2024-09-10T20:25:01.325050+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:01.325050+0000","last_undegraded":"2024-09-10T20:25:01.325050+0000","last_fullsized":"2024-09-10T20:25:01.325050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38804048,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3817,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7342","reported_seq":11262,"reported_epoch":40,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:25:03.347127+0000","last_change":"2024-09-10T20:25:03.051594+0000","last_active":"2024-09-10T20:25:03.347127+0000","last_peered":"2024-09-10T20:25:03.347127+0000","last_clean":"2024-09-10T20:25:03.347127+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:03.347127+0000","last_undegraded":"2024-09-10T20:25:03.347127+0000","last_fullsized":"2024-09-10T20:25:03.347127+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":299,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335072,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7340,"num_write_kb":48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":15589376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7306","reported_seq":7510,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339326+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:01.339326+0000","last_peered":"2024-09-10T20:25:01.339326+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:01.339326+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39374576,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7303,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7367","reported_seq":7582,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.339630+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:01.339630+0000","last_peered":"2024-09-10T20:25:01.339630+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:01.339630+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303760,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7203","reported_seq":7357,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.302341+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:03.302341+0000","last_peered":"2024-09-10T20:25:03.302341+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:03.302341+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37911120,"num_objects":4628,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'7357","reported_seq":7545,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:01.350427+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:01.350427+0000","last_peered":"2024-09-10T20:25:01.350427+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:01.350427+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7357,"log_dups_size":0,"ondisk_log_size":7357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38802592,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7355,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7499","reported_seq":7653,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:03.266587+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:03.266587+0000","last_peered":"2024-09-10T20:25:03.266587+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:03.266587+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547488,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7405","reported_seq":7727,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.590504+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:05.590504+0000","last_peered":"2024-09-10T20:25:05.590504+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:05.590504+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663328,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7403,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":626722984,"num_objects":76509,"num_object_clones":0,"num_object_copies":306036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105198,"num_write_kb":675921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2592,"num_bytes_recovered":16343040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1218797568,"data_stored":1218797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117964,"ondisk_log_size":117964,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705624,"kb_used_data":308800,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93666216,"statfs":{"total":96636764160,"available":95914205184,"internally_reserved":0,"allocated":316211200,"data_stored":316007520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,7,0,1,1,2,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724824,"kb_used_data":306272,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93647016,"statfs":{"total":96636764160,"available":95894544384,"internally_reserved":0,"allocated":313622528,"data_stored":313409467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572188,"kb_used_data":273928,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93799652,"statfs":{"total":96636764160,"available":96050843648,"internally_reserved":0,"allocated":280502272,"data_stored":280289211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305393362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723572,"kb_used_data":304604,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93648268,"statfs":{"total":96636764160,"available":95895826432,"internally_reserved":0,"allocated":311914496,"data_stored":311701435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,4,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310988800,"data_stored":310988800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279920640,"data_stored":279920640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312950784,"data_stored":312950784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314937344,"data_stored":314937344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:10.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:11.409 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:11.409 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:11.434 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-10T20:25:09.993785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":619303944,"num_objects":75603,"num_object_clones":0,"num_object_copies":302412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98742,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":747,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118868,"ondisk_log_size":118868,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730028,"kb_used_data":1183568,"kb_used_omap":14,"kb_used_meta":1546353,"kb_avail":374757332,"statfs":{"total":386547056640,"available":383751507968,"internally_reserved":0,"allocated":1211973632,"data_stored":1211121388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1583466038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,6,11,11,6,0,1,1,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17388704,"num_objects":-2123,"num_object_clones":0,"num_object_copies":-8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-5239,"num_write_kb":-48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1681,"num_bytes_recovered":-14532608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.693113"},"pg_stats":[{"pgid":"1.f","version":"40'7412","reported_seq":7619,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:06.320733+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:06.320733+0000","last_peered":"2024-09-10T20:25:06.320733+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:06.320733+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38584320,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7519","reported_seq":7650,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:09.203601+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:09.203601+0000","last_peered":"2024-09-10T20:25:09.203601+0000","last_clean":"2024-09-10T20:25:09.203601+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:09.203601+0000","last_undegraded":"2024-09-10T20:25:09.203601+0000","last_fullsized":"2024-09-10T20:25:09.203601+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7519,"log_dups_size":0,"ondisk_log_size":7519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38772736,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3960,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7455","reported_seq":7619,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.227578+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:08.227578+0000","last_peered":"2024-09-10T20:25:08.227578+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:08.227578+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38378088,"num_objects":4686,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7454,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7605","reported_seq":11773,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:09.296722+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:09.296722+0000","last_peered":"2024-09-10T20:25:09.296722+0000","last_clean":"2024-09-10T20:25:09.296722+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:09.296722+0000","last_undegraded":"2024-09-10T20:25:09.296722+0000","last_fullsized":"2024-09-10T20:25:09.296722+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7605,"log_dups_size":0,"ondisk_log_size":7605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39329792,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7407","reported_seq":7662,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:06.153597+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:06.153597+0000","last_peered":"2024-09-10T20:25:06.153597+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:06.153597+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7407,"log_dups_size":0,"ondisk_log_size":7407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39007392,"num_objects":4762,"num_object_clones":0,"num_object_copies":19048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7405,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7377","reported_seq":7544,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:06.362578+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:06.362578+0000","last_peered":"2024-09-10T20:25:06.362578+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:06.362578+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7377,"log_dups_size":0,"ondisk_log_size":7377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38674432,"num_objects":4721,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7377,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7360","reported_seq":7508,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.475621+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:05.475621+0000","last_peered":"2024-09-10T20:25:05.475621+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:05.475621+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581408,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7358,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2790}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7527","reported_seq":7658,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.543179+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:05.543179+0000","last_peered":"2024-09-10T20:25:05.543179+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:05.543179+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7527,"log_dups_size":0,"ondisk_log_size":7527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39040160,"num_objects":4766,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7525,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7325","reported_seq":11296,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:06.231370+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:06.231370+0000","last_peered":"2024-09-10T20:25:06.231370+0000","last_clean":"2024-09-10T20:25:06.231370+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:06.231370+0000","last_undegraded":"2024-09-10T20:25:06.231370+0000","last_fullsized":"2024-09-10T20:25:06.231370+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38304336,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3878,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7395","reported_seq":11364,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:08.381773+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:08.381773+0000","last_peered":"2024-09-10T20:25:08.381773+0000","last_clean":"2024-09-10T20:25:08.381773+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:08.381773+0000","last_undegraded":"2024-09-10T20:25:08.381773+0000","last_fullsized":"2024-09-10T20:25:08.381773+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38887424,"num_objects":4747,"num_object_clones":0,"num_object_copies":18988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7377","reported_seq":7599,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:06.076023+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:06.076023+0000","last_peered":"2024-09-10T20:25:06.076023+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:06.076023+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7377,"log_dups_size":0,"ondisk_log_size":7377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38792944,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7434","reported_seq":7665,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:06.302343+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:06.302343+0000","last_peered":"2024-09-10T20:25:06.302343+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:06.302343+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7434,"log_dups_size":0,"ondisk_log_size":7434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38754896,"num_objects":4731,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1894,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7433,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2838}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7269","reported_seq":7427,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.155016+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:08.155016+0000","last_peered":"2024-09-10T20:25:08.155016+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:08.155016+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37370448,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7420","reported_seq":7628,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:06.248280+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:06.248280+0000","last_peered":"2024-09-10T20:25:06.248280+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:06.248280+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7420,"log_dups_size":0,"ondisk_log_size":7420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38286496,"num_objects":4674,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7418,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2769}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7581","reported_seq":7741,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.419326+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:08.419326+0000","last_peered":"2024-09-10T20:25:08.419326+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:08.419326+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875744,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7579,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7405","reported_seq":7727,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:05.590504+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:05.590504+0000","last_peered":"2024-09-10T20:25:05.590504+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:05.590504+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663328,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7403,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":619303944,"num_objects":75603,"num_object_clones":0,"num_object_copies":302412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98742,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":747,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208344576,"data_stored":1208344576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118868,"ondisk_log_size":118868,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117064,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723240,"kb_used_data":304216,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93648600,"statfs":{"total":96636764160,"available":95896166400,"internally_reserved":0,"allocated":311517184,"data_stored":311304123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":429059792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707876,"kb_used_data":301832,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93663964,"statfs":{"total":96636764160,"available":95911899136,"internally_reserved":0,"allocated":309075968,"data_stored":308862907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,6,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575340,"kb_used_data":272916,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93796500,"statfs":{"total":96636764160,"available":96047616000,"internally_reserved":0,"allocated":279465984,"data_stored":279252923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723572,"kb_used_data":304604,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93648268,"statfs":{"total":96636764160,"available":95895826432,"internally_reserved":0,"allocated":311914496,"data_stored":311701435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,4,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310988800,"data_stored":310988800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278814720,"data_stored":278814720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308273152,"data_stored":308273152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310267904,"data_stored":310267904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:11.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:12.033 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-10T20:25:11.994281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612799496,"num_objects":74809,"num_object_clones":0,"num_object_copies":299236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99536,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119662,"ondisk_log_size":119662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695316,"kb_used_data":1173824,"kb_used_omap":14,"kb_used_meta":1521393,"kb_avail":374792044,"statfs":{"total":386547056640,"available":383787053056,"internally_reserved":0,"allocated":1201995776,"data_stored":1201143532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1557906998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,8,8,24,9,0,0,0,1,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19494048,"num_objects":-2380,"num_object_clones":0,"num_object_copies":-9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-4982,"num_write_kb":-48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1630,"num_bytes_recovered":-14131200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003013"},"pg_stats":[{"pgid":"1.f","version":"40'7484","reported_seq":7701,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.370564+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:11.370564+0000","last_peered":"2024-09-10T20:25:11.370564+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:11.370564+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7484,"log_dups_size":0,"ondisk_log_size":7484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37994496,"num_objects":4638,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7484,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7519","reported_seq":7650,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:09.203601+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:09.203601+0000","last_peered":"2024-09-10T20:25:09.203601+0000","last_clean":"2024-09-10T20:25:09.203601+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:09.203601+0000","last_undegraded":"2024-09-10T20:25:09.203601+0000","last_fullsized":"2024-09-10T20:25:09.203601+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7519,"log_dups_size":0,"ondisk_log_size":7519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38772736,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3960,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7455","reported_seq":7619,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.227578+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:08.227578+0000","last_peered":"2024-09-10T20:25:08.227578+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:08.227578+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38378088,"num_objects":4686,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7454,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7605","reported_seq":11773,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:09.296722+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:09.296722+0000","last_peered":"2024-09-10T20:25:09.296722+0000","last_clean":"2024-09-10T20:25:09.296722+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:09.296722+0000","last_undegraded":"2024-09-10T20:25:09.296722+0000","last_fullsized":"2024-09-10T20:25:09.296722+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7605,"log_dups_size":0,"ondisk_log_size":7605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39329792,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7488","reported_seq":7749,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.335288+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:11.335288+0000","last_peered":"2024-09-10T20:25:11.335288+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:11.335288+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38343840,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7486,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7467","reported_seq":7640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.300232+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:11.300232+0000","last_peered":"2024-09-10T20:25:11.300232+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:11.300232+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7467,"log_dups_size":0,"ondisk_log_size":7467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37937152,"num_objects":4631,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7427","reported_seq":7579,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:10.503159+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:10.503159+0000","last_peered":"2024-09-10T20:25:10.503159+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:10.503159+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38032544,"num_objects":4643,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7425,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7616","reported_seq":7747,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:10.490560+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:10.490560+0000","last_peered":"2024-09-10T20:25:10.490560+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:10.490560+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7616,"log_dups_size":0,"ondisk_log_size":7616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311072,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7614,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7393","reported_seq":11363,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:11.117191+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:11.117191+0000","last_peered":"2024-09-10T20:25:11.117191+0000","last_clean":"2024-09-10T20:25:11.117191+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:11.117191+0000","last_undegraded":"2024-09-10T20:25:11.117191+0000","last_fullsized":"2024-09-10T20:25:11.117191+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37747280,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7395","reported_seq":11364,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:08.381773+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:08.381773+0000","last_peered":"2024-09-10T20:25:08.381773+0000","last_clean":"2024-09-10T20:25:08.381773+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:08.381773+0000","last_undegraded":"2024-09-10T20:25:08.381773+0000","last_fullsized":"2024-09-10T20:25:08.381773+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38887424,"num_objects":4747,"num_object_clones":0,"num_object_copies":18988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7454","reported_seq":7686,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.262501+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:11.262501+0000","last_peered":"2024-09-10T20:25:11.262501+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:11.262501+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38162160,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1879,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7451,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2783}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7504","reported_seq":7745,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.166974+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:11.166974+0000","last_peered":"2024-09-10T20:25:11.166974+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:11.166974+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38181456,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7269","reported_seq":7427,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.155016+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:08.155016+0000","last_peered":"2024-09-10T20:25:08.155016+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:08.155016+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37370448,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7510","reported_seq":7726,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.413520+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:11.413520+0000","last_peered":"2024-09-10T20:25:11.413520+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:11.413520+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37549216,"num_objects":4584,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7508,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7581","reported_seq":7741,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.419326+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:08.419326+0000","last_peered":"2024-09-10T20:25:08.419326+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:08.419326+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875744,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7579,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7495","reported_seq":7915,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:10.596323+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:10.596323+0000","last_peered":"2024-09-10T20:25:10.596323+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:10.596323+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7495,"log_dups_size":0,"ondisk_log_size":7495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926048,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7493,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":612799496,"num_objects":74809,"num_object_clones":0,"num_object_copies":299236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99536,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198051328,"data_stored":1198051328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119662,"ondisk_log_size":119662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705796,"kb_used_data":299252,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93666044,"statfs":{"total":96636764160,"available":95914029056,"internally_reserved":0,"allocated":306434048,"data_stored":306220987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":416280272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,11,3,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707876,"kb_used_data":301832,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93663964,"statfs":{"total":96636764160,"available":95911899136,"internally_reserved":0,"allocated":309075968,"data_stored":308862907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,6,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575340,"kb_used_data":272916,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93796500,"statfs":{"total":96636764160,"available":96047616000,"internally_reserved":0,"allocated":279465984,"data_stored":279252923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706304,"kb_used_data":299824,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93665536,"statfs":{"total":96636764160,"available":95913508864,"internally_reserved":0,"allocated":307019776,"data_stored":306806715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305876992,"data_stored":305876992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278814720,"data_stored":278814720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308273152,"data_stored":308273152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305086464,"data_stored":305086464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:12.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:12.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'7484", 'reported_seq': 7701, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.370564+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:11.370564+0000', 'last_peered': '2024-09-10T20:25:11.370564+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:11.370564+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7484, 'log_dups_size': 0, 'ondisk_log_size': 7484, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37994496, 'num_objects': 4638, 'num_object_clones': 0, 'num_object_copies': 18552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1887, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7484, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:12.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7455", 'reported_seq': 7619, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:08.227578+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:08.227578+0000', 'last_peered': '2024-09-10T20:25:08.227578+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:08.227578+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7455, 'log_dups_size': 0, 'ondisk_log_size': 7455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38378088, 'num_objects': 4686, 'num_object_clones': 0, 'num_object_copies': 18744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1928, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4686, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7454, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2759}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'7488", 'reported_seq': 7749, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.335288+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:11.335288+0000', 'last_peered': '2024-09-10T20:25:11.335288+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:11.335288+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7488, 'log_dups_size': 0, 'ondisk_log_size': 7488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38343840, 'num_objects': 4681, 'num_object_clones': 0, 'num_object_copies': 18724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1886, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7486, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2797}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7467", 'reported_seq': 7640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.300232+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:11.300232+0000', 'last_peered': '2024-09-10T20:25:11.300232+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:11.300232+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7467, 'log_dups_size': 0, 'ondisk_log_size': 7467, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37937152, 'num_objects': 4631, 'num_object_clones': 0, 'num_object_copies': 18524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7467, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 57, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2750}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7427", 'reported_seq': 7579, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:10.503159+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:10.503159+0000', 'last_peered': '2024-09-10T20:25:10.503159+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:10.503159+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7427, 'log_dups_size': 0, 'ondisk_log_size': 7427, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38032544, 'num_objects': 4643, 'num_object_clones': 0, 'num_object_copies': 18572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1920, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7425, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2725}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'7616", 'reported_seq': 7747, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:10.490560+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:10.490560+0000', 'last_peered': '2024-09-10T20:25:10.490560+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:10.490560+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7616, 'log_dups_size': 0, 'ondisk_log_size': 7616, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38311072, 'num_objects': 4677, 'num_object_clones': 0, 'num_object_copies': 18708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7614, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2659}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7454", 'reported_seq': 7686, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.262501+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:11.262501+0000', 'last_peered': '2024-09-10T20:25:11.262501+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:11.262501+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7454, 'log_dups_size': 0, 'ondisk_log_size': 7454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38162160, 'num_objects': 4659, 'num_object_clones': 0, 'num_object_copies': 18636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1879, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7451, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2783}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:12.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'7504", 'reported_seq': 7745, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.166974+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:11.166974+0000', 'last_peered': '2024-09-10T20:25:11.166974+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:11.166974+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7504, 'log_dups_size': 0, 'ondisk_log_size': 7504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38181456, 'num_objects': 4661, 'num_object_clones': 0, 'num_object_copies': 18644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1889, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7503, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7269", 'reported_seq': 7427, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:08.155016+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:08.155016+0000', 'last_peered': '2024-09-10T20:25:08.155016+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:08.155016+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7269, 'log_dups_size': 0, 'ondisk_log_size': 7269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37370448, 'num_objects': 4562, 'num_object_clones': 0, 'num_object_copies': 18248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7268, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2647}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'7510", 'reported_seq': 7726, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.413520+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:11.413520+0000', 'last_peered': '2024-09-10T20:25:11.413520+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:11.413520+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7510, 'log_dups_size': 0, 'ondisk_log_size': 7510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37549216, 'num_objects': 4584, 'num_object_clones': 0, 'num_object_copies': 18336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1903, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7508, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2683}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'7581", 'reported_seq': 7741, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:08.419326+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:08.419326+0000', 'last_peered': '2024-09-10T20:25:08.419326+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:08.419326+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7581, 'log_dups_size': 0, 'ondisk_log_size': 7581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39875744, 'num_objects': 4868, 'num_object_clones': 0, 'num_object_copies': 19472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1932, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7579, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:12.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7495", 'reported_seq': 7915, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:10.596323+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:10.596323+0000', 'last_peered': '2024-09-10T20:25:10.596323+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:10.596323+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7495, 'log_dups_size': 0, 'ondisk_log_size': 7495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37926048, 'num_objects': 4630, 'num_object_clones': 0, 'num_object_copies': 18520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1812, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7493, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 171, 'num_bytes_recovered': 1155072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:12.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:12.764 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:12.764 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22853/299236 objects degraded (7.637%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":401,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74809,"data_bytes":612799496,"bytes_used":2760003584,"bytes_avail":383787053056,"bytes_total":386547056640,"degraded_objects":22853,"degraded_total":299236,"degraded_ratio":0.076371158550441798},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:12.783 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:25:12.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:13.365 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:13.365 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:13.403 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-10T20:25:11.994281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612799496,"num_objects":74809,"num_object_clones":0,"num_object_copies":299236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99536,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119662,"ondisk_log_size":119662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695316,"kb_used_data":1173824,"kb_used_omap":14,"kb_used_meta":1521393,"kb_avail":374792044,"statfs":{"total":386547056640,"available":383787053056,"internally_reserved":0,"allocated":1201995776,"data_stored":1201143532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1557906998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,8,8,24,9,0,0,0,1,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19494048,"num_objects":-2380,"num_object_clones":0,"num_object_copies":-9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-4982,"num_write_kb":-48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1630,"num_bytes_recovered":-14131200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003013"},"pg_stats":[{"pgid":"1.f","version":"40'7484","reported_seq":7701,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.370564+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:11.370564+0000","last_peered":"2024-09-10T20:25:11.370564+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:11.370564+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7484,"log_dups_size":0,"ondisk_log_size":7484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37994496,"num_objects":4638,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7484,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7519","reported_seq":7650,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:09.203601+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:09.203601+0000","last_peered":"2024-09-10T20:25:09.203601+0000","last_clean":"2024-09-10T20:25:09.203601+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:09.203601+0000","last_undegraded":"2024-09-10T20:25:09.203601+0000","last_fullsized":"2024-09-10T20:25:09.203601+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7519,"log_dups_size":0,"ondisk_log_size":7519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38772736,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3960,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7455","reported_seq":7619,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.227578+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:08.227578+0000","last_peered":"2024-09-10T20:25:08.227578+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:08.227578+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38378088,"num_objects":4686,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7454,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7605","reported_seq":11773,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:09.296722+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:09.296722+0000","last_peered":"2024-09-10T20:25:09.296722+0000","last_clean":"2024-09-10T20:25:09.296722+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:09.296722+0000","last_undegraded":"2024-09-10T20:25:09.296722+0000","last_fullsized":"2024-09-10T20:25:09.296722+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7605,"log_dups_size":0,"ondisk_log_size":7605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39329792,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7488","reported_seq":7749,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.335288+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:11.335288+0000","last_peered":"2024-09-10T20:25:11.335288+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:11.335288+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38343840,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7486,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7467","reported_seq":7640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.300232+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:11.300232+0000","last_peered":"2024-09-10T20:25:11.300232+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:11.300232+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7467,"log_dups_size":0,"ondisk_log_size":7467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37937152,"num_objects":4631,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7427","reported_seq":7579,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:10.503159+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:10.503159+0000","last_peered":"2024-09-10T20:25:10.503159+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:10.503159+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38032544,"num_objects":4643,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7425,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7616","reported_seq":7747,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:10.490560+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:10.490560+0000","last_peered":"2024-09-10T20:25:10.490560+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:10.490560+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7616,"log_dups_size":0,"ondisk_log_size":7616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311072,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7614,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7393","reported_seq":11363,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:11.117191+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:11.117191+0000","last_peered":"2024-09-10T20:25:11.117191+0000","last_clean":"2024-09-10T20:25:11.117191+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:11.117191+0000","last_undegraded":"2024-09-10T20:25:11.117191+0000","last_fullsized":"2024-09-10T20:25:11.117191+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37747280,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7395","reported_seq":11364,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:08.381773+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:08.381773+0000","last_peered":"2024-09-10T20:25:08.381773+0000","last_clean":"2024-09-10T20:25:08.381773+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:08.381773+0000","last_undegraded":"2024-09-10T20:25:08.381773+0000","last_fullsized":"2024-09-10T20:25:08.381773+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38887424,"num_objects":4747,"num_object_clones":0,"num_object_copies":18988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7454","reported_seq":7686,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.262501+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:11.262501+0000","last_peered":"2024-09-10T20:25:11.262501+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:11.262501+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38162160,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1879,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7451,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2783}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7504","reported_seq":7745,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.166974+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:11.166974+0000","last_peered":"2024-09-10T20:25:11.166974+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:11.166974+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38181456,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7269","reported_seq":7427,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.155016+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:08.155016+0000","last_peered":"2024-09-10T20:25:08.155016+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:08.155016+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37370448,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7510","reported_seq":7726,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.413520+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:11.413520+0000","last_peered":"2024-09-10T20:25:11.413520+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:11.413520+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37549216,"num_objects":4584,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7508,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7581","reported_seq":7741,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:08.419326+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:08.419326+0000","last_peered":"2024-09-10T20:25:08.419326+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:08.419326+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875744,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7579,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7495","reported_seq":7915,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:10.596323+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:10.596323+0000","last_peered":"2024-09-10T20:25:10.596323+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:10.596323+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7495,"log_dups_size":0,"ondisk_log_size":7495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926048,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7493,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":612799496,"num_objects":74809,"num_object_clones":0,"num_object_copies":299236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99536,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198051328,"data_stored":1198051328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119662,"ondisk_log_size":119662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705796,"kb_used_data":299252,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93666044,"statfs":{"total":96636764160,"available":95914029056,"internally_reserved":0,"allocated":306434048,"data_stored":306220987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":416280272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,11,3,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707876,"kb_used_data":301832,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93663964,"statfs":{"total":96636764160,"available":95911899136,"internally_reserved":0,"allocated":309075968,"data_stored":308862907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,6,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575340,"kb_used_data":272916,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93796500,"statfs":{"total":96636764160,"available":96047616000,"internally_reserved":0,"allocated":279465984,"data_stored":279252923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706304,"kb_used_data":299824,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93665536,"statfs":{"total":96636764160,"available":95913508864,"internally_reserved":0,"allocated":307019776,"data_stored":306806715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305876992,"data_stored":305876992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278814720,"data_stored":278814720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308273152,"data_stored":308273152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305086464,"data_stored":305086464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:16.405 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:16.966 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:16.990 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-10T20:25:15.995940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606917640,"num_objects":74091,"num_object_clones":0,"num_object_copies":296364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100254,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":884,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120380,"ondisk_log_size":120380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732332,"kb_used_data":1162256,"kb_used_omap":14,"kb_used_meta":1569969,"kb_avail":374755028,"statfs":{"total":386547056640,"available":383749148672,"internally_reserved":0,"allocated":1190150144,"data_stored":1189297900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1607648822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,7,23,8,1,0,0,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23082144,"num_objects":-2818,"num_object_clones":0,"num_object_copies":-11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-4544,"num_write_kb":-48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1607,"num_bytes_recovered":-13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004004"},"pg_stats":[{"pgid":"1.f","version":"40'7484","reported_seq":7701,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.370564+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:11.370564+0000","last_peered":"2024-09-10T20:25:11.370564+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:11.370564+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7484,"log_dups_size":0,"ondisk_log_size":7484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37994496,"num_objects":4638,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7484,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7601","reported_seq":7732,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:14.059334+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:14.059334+0000","last_peered":"2024-09-10T20:25:14.059334+0000","last_clean":"2024-09-10T20:25:14.059334+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:14.059334+0000","last_undegraded":"2024-09-10T20:25:14.059334+0000","last_fullsized":"2024-09-10T20:25:14.059334+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7601,"log_dups_size":0,"ondisk_log_size":7601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4042,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7528","reported_seq":7695,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.440437+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:13.440437+0000","last_peered":"2024-09-10T20:25:13.440437+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:13.440437+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7528,"log_dups_size":0,"ondisk_log_size":7528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37780072,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7527,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2688}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7689","reported_seq":11857,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:14.327234+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:14.327234+0000","last_peered":"2024-09-10T20:25:14.327234+0000","last_clean":"2024-09-10T20:25:14.327234+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:14.327234+0000","last_undegraded":"2024-09-10T20:25:14.327234+0000","last_fullsized":"2024-09-10T20:25:14.327234+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7689,"log_dups_size":0,"ondisk_log_size":7689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1949,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7488","reported_seq":7749,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.335288+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:11.335288+0000","last_peered":"2024-09-10T20:25:11.335288+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:11.335288+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38343840,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7486,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7467","reported_seq":7640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.300232+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:11.300232+0000","last_peered":"2024-09-10T20:25:11.300232+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:11.300232+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7467,"log_dups_size":0,"ondisk_log_size":7467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37937152,"num_objects":4631,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7516","reported_seq":7671,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.546304+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:15.546304+0000","last_peered":"2024-09-10T20:25:15.546304+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:15.546304+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37303456,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7514,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2639}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7692","reported_seq":7823,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.658976+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:15.658976+0000","last_peered":"2024-09-10T20:25:15.658976+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:15.658976+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688480,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7393","reported_seq":11363,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:11.117191+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:11.117191+0000","last_peered":"2024-09-10T20:25:11.117191+0000","last_clean":"2024-09-10T20:25:11.117191+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:11.117191+0000","last_undegraded":"2024-09-10T20:25:11.117191+0000","last_fullsized":"2024-09-10T20:25:11.117191+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37747280,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7476","reported_seq":11445,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:13.380880+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:13.380880+0000","last_peered":"2024-09-10T20:25:13.380880+0000","last_clean":"2024-09-10T20:25:13.380880+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:13.380880+0000","last_undegraded":"2024-09-10T20:25:13.380880+0000","last_fullsized":"2024-09-10T20:25:13.380880+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7476,"log_dups_size":0,"ondisk_log_size":7476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38223872,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7454","reported_seq":7686,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.262501+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:11.262501+0000","last_peered":"2024-09-10T20:25:11.262501+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:11.262501+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38162160,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1879,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7451,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2783}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7504","reported_seq":7745,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.166974+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:11.166974+0000","last_peered":"2024-09-10T20:25:11.166974+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:11.166974+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38181456,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7347","reported_seq":7505,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.187272+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:13.187272+0000","last_peered":"2024-09-10T20:25:13.187272+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:13.187272+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36731472,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2569}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7510","reported_seq":7726,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.413520+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:11.413520+0000","last_peered":"2024-09-10T20:25:11.413520+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:11.413520+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37549216,"num_objects":4584,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7508,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7669","reported_seq":7831,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.376440+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:13.376440+0000","last_peered":"2024-09-10T20:25:13.376440+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:13.376440+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39154848,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7562","reported_seq":8089,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.686240+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:15.686240+0000","last_peered":"2024-09-10T20:25:15.686240+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:15.686240+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37377184,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606917640,"num_objects":74091,"num_object_clones":0,"num_object_copies":296364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100254,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":884,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1186304000,"data_stored":1186304000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120380,"ondisk_log_size":120380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705796,"kb_used_data":299252,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93666044,"statfs":{"total":96636764160,"available":95914029056,"internally_reserved":0,"allocated":306434048,"data_stored":306220987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":416280272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,11,3,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724944,"kb_used_data":296696,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93646896,"statfs":{"total":96636764160,"available":95894421504,"internally_reserved":0,"allocated":303816704,"data_stored":303603643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578236,"kb_used_data":271652,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93793604,"statfs":{"total":96636764160,"available":96044650496,"internally_reserved":0,"allocated":278171648,"data_stored":277958587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723356,"kb_used_data":294656,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93648484,"statfs":{"total":96636764160,"available":95896047616,"internally_reserved":0,"allocated":301727744,"data_stored":301514683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300572672,"data_stored":300572672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277553152,"data_stored":277553152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303091712,"data_stored":303091712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305086464,"data_stored":305086464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:16.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:17.642 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:17.666 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-10T20:25:15.995940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606917640,"num_objects":74091,"num_object_clones":0,"num_object_copies":296364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100254,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":884,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120380,"ondisk_log_size":120380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732332,"kb_used_data":1162256,"kb_used_omap":14,"kb_used_meta":1569969,"kb_avail":374755028,"statfs":{"total":386547056640,"available":383749148672,"internally_reserved":0,"allocated":1190150144,"data_stored":1189297900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1607648822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,7,23,8,1,0,0,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23082144,"num_objects":-2818,"num_object_clones":0,"num_object_copies":-11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-4544,"num_write_kb":-48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1607,"num_bytes_recovered":-13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004004"},"pg_stats":[{"pgid":"1.f","version":"40'7484","reported_seq":7701,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.370564+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:11.370564+0000","last_peered":"2024-09-10T20:25:11.370564+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:11.370564+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7484,"log_dups_size":0,"ondisk_log_size":7484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37994496,"num_objects":4638,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7484,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7601","reported_seq":7732,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:14.059334+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:14.059334+0000","last_peered":"2024-09-10T20:25:14.059334+0000","last_clean":"2024-09-10T20:25:14.059334+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:14.059334+0000","last_undegraded":"2024-09-10T20:25:14.059334+0000","last_fullsized":"2024-09-10T20:25:14.059334+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7601,"log_dups_size":0,"ondisk_log_size":7601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4042,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7528","reported_seq":7695,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.440437+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:13.440437+0000","last_peered":"2024-09-10T20:25:13.440437+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:13.440437+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7528,"log_dups_size":0,"ondisk_log_size":7528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37780072,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7527,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2688}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7689","reported_seq":11857,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:14.327234+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:14.327234+0000","last_peered":"2024-09-10T20:25:14.327234+0000","last_clean":"2024-09-10T20:25:14.327234+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:14.327234+0000","last_undegraded":"2024-09-10T20:25:14.327234+0000","last_fullsized":"2024-09-10T20:25:14.327234+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7689,"log_dups_size":0,"ondisk_log_size":7689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1949,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7488","reported_seq":7749,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.335288+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:11.335288+0000","last_peered":"2024-09-10T20:25:11.335288+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:11.335288+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38343840,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7486,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7467","reported_seq":7640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.300232+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:11.300232+0000","last_peered":"2024-09-10T20:25:11.300232+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:11.300232+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7467,"log_dups_size":0,"ondisk_log_size":7467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37937152,"num_objects":4631,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7516","reported_seq":7671,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.546304+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:15.546304+0000","last_peered":"2024-09-10T20:25:15.546304+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:15.546304+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37303456,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7514,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2639}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7692","reported_seq":7823,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.658976+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:15.658976+0000","last_peered":"2024-09-10T20:25:15.658976+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:15.658976+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688480,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7393","reported_seq":11363,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:11.117191+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:11.117191+0000","last_peered":"2024-09-10T20:25:11.117191+0000","last_clean":"2024-09-10T20:25:11.117191+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:11.117191+0000","last_undegraded":"2024-09-10T20:25:11.117191+0000","last_fullsized":"2024-09-10T20:25:11.117191+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37747280,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7476","reported_seq":11445,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:13.380880+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:13.380880+0000","last_peered":"2024-09-10T20:25:13.380880+0000","last_clean":"2024-09-10T20:25:13.380880+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:13.380880+0000","last_undegraded":"2024-09-10T20:25:13.380880+0000","last_fullsized":"2024-09-10T20:25:13.380880+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7476,"log_dups_size":0,"ondisk_log_size":7476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38223872,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7454","reported_seq":7686,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.262501+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:11.262501+0000","last_peered":"2024-09-10T20:25:11.262501+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:11.262501+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38162160,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1879,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7451,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2783}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7504","reported_seq":7745,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.166974+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:11.166974+0000","last_peered":"2024-09-10T20:25:11.166974+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:11.166974+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38181456,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7347","reported_seq":7505,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.187272+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:13.187272+0000","last_peered":"2024-09-10T20:25:13.187272+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:13.187272+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36731472,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2569}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7510","reported_seq":7726,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:11.413520+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:11.413520+0000","last_peered":"2024-09-10T20:25:11.413520+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:11.413520+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37549216,"num_objects":4584,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7508,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7669","reported_seq":7831,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.376440+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:13.376440+0000","last_peered":"2024-09-10T20:25:13.376440+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:13.376440+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39154848,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7562","reported_seq":8089,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.686240+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:15.686240+0000","last_peered":"2024-09-10T20:25:15.686240+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:15.686240+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37377184,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606917640,"num_objects":74091,"num_object_clones":0,"num_object_copies":296364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100254,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":884,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1186304000,"data_stored":1186304000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120380,"ondisk_log_size":120380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705796,"kb_used_data":299252,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93666044,"statfs":{"total":96636764160,"available":95914029056,"internally_reserved":0,"allocated":306434048,"data_stored":306220987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":416280272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,11,3,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724944,"kb_used_data":296696,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93646896,"statfs":{"total":96636764160,"available":95894421504,"internally_reserved":0,"allocated":303816704,"data_stored":303603643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578236,"kb_used_data":271652,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93793604,"statfs":{"total":96636764160,"available":96044650496,"internally_reserved":0,"allocated":278171648,"data_stored":277958587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723356,"kb_used_data":294656,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93648484,"statfs":{"total":96636764160,"available":95896047616,"internally_reserved":0,"allocated":301727744,"data_stored":301514683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300572672,"data_stored":300572672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277553152,"data_stored":277553152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303091712,"data_stored":303091712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305086464,"data_stored":305086464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:17.667 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'7484", 'reported_seq': 7701, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.370564+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:11.370564+0000', 'last_peered': '2024-09-10T20:25:11.370564+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:11.370564+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7484, 'log_dups_size': 0, 'ondisk_log_size': 7484, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37994496, 'num_objects': 4638, 'num_object_clones': 0, 'num_object_copies': 18552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1887, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7484, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7528", 'reported_seq': 7695, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:13.440437+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:13.440437+0000', 'last_peered': '2024-09-10T20:25:13.440437+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:13.440437+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7528, 'log_dups_size': 0, 'ondisk_log_size': 7528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37780072, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1926, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7527, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2688}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'7488", 'reported_seq': 7749, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.335288+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:11.335288+0000', 'last_peered': '2024-09-10T20:25:11.335288+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:11.335288+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7488, 'log_dups_size': 0, 'ondisk_log_size': 7488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38343840, 'num_objects': 4681, 'num_object_clones': 0, 'num_object_copies': 18724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1886, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7486, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2797}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7467", 'reported_seq': 7640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.300232+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:11.300232+0000', 'last_peered': '2024-09-10T20:25:11.300232+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:11.300232+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7467, 'log_dups_size': 0, 'ondisk_log_size': 7467, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37937152, 'num_objects': 4631, 'num_object_clones': 0, 'num_object_copies': 18524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7467, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 57, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2750}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7516", 'reported_seq': 7671, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:15.546304+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:15.546304+0000', 'last_peered': '2024-09-10T20:25:15.546304+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:15.546304+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7516, 'log_dups_size': 0, 'ondisk_log_size': 7516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37303456, 'num_objects': 4554, 'num_object_clones': 0, 'num_object_copies': 18216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1917, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7514, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2639}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:17.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'7692", 'reported_seq': 7823, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:15.658976+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:15.658976+0000', 'last_peered': '2024-09-10T20:25:15.658976+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:15.658976+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7692, 'log_dups_size': 0, 'ondisk_log_size': 7692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37688480, 'num_objects': 4601, 'num_object_clones': 0, 'num_object_copies': 18404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7690, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7454", 'reported_seq': 7686, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.262501+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:11.262501+0000', 'last_peered': '2024-09-10T20:25:11.262501+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:11.262501+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7454, 'log_dups_size': 0, 'ondisk_log_size': 7454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38162160, 'num_objects': 4659, 'num_object_clones': 0, 'num_object_copies': 18636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1879, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7451, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2783}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'7504", 'reported_seq': 7745, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.166974+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:11.166974+0000', 'last_peered': '2024-09-10T20:25:11.166974+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:11.166974+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7504, 'log_dups_size': 0, 'ondisk_log_size': 7504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38181456, 'num_objects': 4661, 'num_object_clones': 0, 'num_object_copies': 18644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1889, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7503, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7347", 'reported_seq': 7505, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:13.187272+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:13.187272+0000', 'last_peered': '2024-09-10T20:25:13.187272+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:13.187272+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7347, 'log_dups_size': 0, 'ondisk_log_size': 7347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36731472, 'num_objects': 4484, 'num_object_clones': 0, 'num_object_copies': 17936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7346, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'7510", 'reported_seq': 7726, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:11.413520+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:11.413520+0000', 'last_peered': '2024-09-10T20:25:11.413520+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:11.413520+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7510, 'log_dups_size': 0, 'ondisk_log_size': 7510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37549216, 'num_objects': 4584, 'num_object_clones': 0, 'num_object_copies': 18336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1903, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7508, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2683}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:17.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'7669", 'reported_seq': 7831, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:13.376440+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:13.376440+0000', 'last_peered': '2024-09-10T20:25:13.376440+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:13.376440+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7669, 'log_dups_size': 0, 'ondisk_log_size': 7669, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39154848, 'num_objects': 4780, 'num_object_clones': 0, 'num_object_copies': 19120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7667, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:17.670 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:17.670 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7562", 'reported_seq': 8089, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:15.686240+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:15.686240+0000', 'last_peered': '2024-09-10T20:25:15.686240+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:15.686240+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7562, 'log_dups_size': 0, 'ondisk_log_size': 7562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37377184, 'num_objects': 4563, 'num_object_clones': 0, 'num_object_copies': 18252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1758, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7560, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 225, 'num_bytes_recovered': 1597440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:17.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:18.369 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:18.370 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22792/296364 objects degraded (7.691%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":407,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74091,"data_bytes":606917640,"bytes_used":2797907968,"bytes_avail":383749148672,"bytes_total":386547056640,"degraded_objects":22792,"degraded_total":296364,"degraded_ratio":0.076905427109905389},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:18.393 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:25:18.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:18.958 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:18.959 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:18.980 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-10T20:25:17.996806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601969672,"num_objects":73487,"num_object_clones":0,"num_object_copies":293948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73487,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100858,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120984,"ondisk_log_size":120984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749212,"kb_used_data":1156928,"kb_used_omap":14,"kb_used_meta":1592177,"kb_avail":374738148,"statfs":{"total":386547056640,"available":383731863552,"internally_reserved":0,"allocated":1184694272,"data_stored":1183842028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1630389814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,12,18,7,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24753312,"num_objects":-3022,"num_object_clones":0,"num_object_copies":-12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-4340,"num_write_kb":-48566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1671,"num_bytes_recovered":-14745600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004175"},"pg_stats":[{"pgid":"1.f","version":"40'7569","reported_seq":7800,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:16.407985+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:16.407985+0000","last_peered":"2024-09-10T20:25:16.407985+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:16.407985+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37298176,"num_objects":4553,"num_object_clones":0,"num_object_copies":18212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1879,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7601","reported_seq":7732,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:14.059334+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:14.059334+0000","last_peered":"2024-09-10T20:25:14.059334+0000","last_clean":"2024-09-10T20:25:14.059334+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:14.059334+0000","last_undegraded":"2024-09-10T20:25:14.059334+0000","last_fullsized":"2024-09-10T20:25:14.059334+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7601,"log_dups_size":0,"ondisk_log_size":7601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4042,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7528","reported_seq":7695,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.440437+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:13.440437+0000","last_peered":"2024-09-10T20:25:13.440437+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:13.440437+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7528,"log_dups_size":0,"ondisk_log_size":7528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37780072,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7527,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2688}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7689","reported_seq":11857,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:14.327234+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:14.327234+0000","last_peered":"2024-09-10T20:25:14.327234+0000","last_clean":"2024-09-10T20:25:14.327234+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:14.327234+0000","last_undegraded":"2024-09-10T20:25:14.327234+0000","last_fullsized":"2024-09-10T20:25:14.327234+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7689,"log_dups_size":0,"ondisk_log_size":7689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1949,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7568","reported_seq":7839,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:16.335610+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:16.335610+0000","last_peered":"2024-09-10T20:25:16.335610+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:16.335610+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688480,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7549","reported_seq":7736,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:16.426248+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:16.426248+0000","last_peered":"2024-09-10T20:25:16.426248+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:16.426248+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37265408,"num_objects":4549,"num_object_clones":0,"num_object_copies":18196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2675}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7516","reported_seq":7671,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.546304+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:15.546304+0000","last_peered":"2024-09-10T20:25:15.546304+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:15.546304+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37303456,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7514,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2639}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7692","reported_seq":7823,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.658976+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:15.658976+0000","last_peered":"2024-09-10T20:25:15.658976+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:15.658976+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688480,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7499","reported_seq":11470,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:16.297373+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:16.297373+0000","last_peered":"2024-09-10T20:25:16.297373+0000","last_clean":"2024-09-10T20:25:16.297373+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:16.297373+0000","last_undegraded":"2024-09-10T20:25:16.297373+0000","last_fullsized":"2024-09-10T20:25:16.297373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36878928,"num_objects":4502,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4052,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7476","reported_seq":11445,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:13.380880+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:13.380880+0000","last_peered":"2024-09-10T20:25:13.380880+0000","last_clean":"2024-09-10T20:25:13.380880+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:13.380880+0000","last_undegraded":"2024-09-10T20:25:13.380880+0000","last_fullsized":"2024-09-10T20:25:13.380880+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7476,"log_dups_size":0,"ondisk_log_size":7476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38223872,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7537","reported_seq":7775,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:16.401111+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:16.401111+0000","last_peered":"2024-09-10T20:25:16.401111+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:16.401111+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37482224,"num_objects":4576,"num_object_clones":0,"num_object_copies":18304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7534,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2703}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7594","reported_seq":7851,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:16.352194+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:16.352194+0000","last_peered":"2024-09-10T20:25:16.352194+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:16.352194+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7594,"log_dups_size":0,"ondisk_log_size":7594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37444176,"num_objects":4571,"num_object_clones":0,"num_object_copies":18284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2691}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7347","reported_seq":7505,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.187272+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:13.187272+0000","last_peered":"2024-09-10T20:25:13.187272+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:13.187272+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36731472,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2569}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7588","reported_seq":7816,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:16.367333+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:16.367333+0000","last_peered":"2024-09-10T20:25:16.367333+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:16.367333+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7588,"log_dups_size":0,"ondisk_log_size":7588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36910240,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7586,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2611}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7669","reported_seq":7831,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:13.376440+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:13.376440+0000","last_peered":"2024-09-10T20:25:13.376440+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:13.376440+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39154848,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7562","reported_seq":8089,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:15.686240+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:15.686240+0000","last_peered":"2024-09-10T20:25:15.686240+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:15.686240+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37377184,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":601969672,"num_objects":73487,"num_object_clones":0,"num_object_copies":293948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73487,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100858,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1180983296,"data_stored":1180983296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120984,"ondisk_log_size":120984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117066,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722676,"kb_used_data":293924,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93649164,"statfs":{"total":96636764160,"available":95896743936,"internally_reserved":0,"allocated":300978176,"data_stored":300765115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724944,"kb_used_data":296696,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93646896,"statfs":{"total":96636764160,"available":95894421504,"internally_reserved":0,"allocated":303816704,"data_stored":303603643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578236,"kb_used_data":271652,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93793604,"statfs":{"total":96636764160,"available":96044650496,"internally_reserved":0,"allocated":278171648,"data_stored":277958587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723356,"kb_used_data":294656,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93648484,"statfs":{"total":96636764160,"available":95896047616,"internally_reserved":0,"allocated":301727744,"data_stored":301514683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300572672,"data_stored":300572672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277553152,"data_stored":277553152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303091712,"data_stored":303091712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":299765760,"data_stored":299765760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:21.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:22.580 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:22.608 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-10T20:25:21.998509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591852552,"num_objects":72252,"num_object_clones":0,"num_object_copies":289008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102093,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1026,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122219,"ondisk_log_size":122219,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748556,"kb_used_data":1139636,"kb_used_omap":14,"kb_used_meta":1608817,"kb_avail":374738804,"statfs":{"total":386547056640,"available":383732535296,"internally_reserved":0,"allocated":1166987264,"data_stored":1166135020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1647429174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,10,13,11,2,0,2,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27451392,"num_objects":-3351,"num_object_clones":0,"num_object_copies":-13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004723"},"pg_stats":[{"pgid":"1.f","version":"40'7643","reported_seq":7890,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.310636+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:21.310636+0000","last_peered":"2024-09-10T20:25:21.310636+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:21.310636+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7643,"log_dups_size":0,"ondisk_log_size":7643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36691968,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7643,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7687","reported_seq":7818,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:19.306186+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:19.306186+0000","last_peered":"2024-09-10T20:25:19.306186+0000","last_clean":"2024-09-10T20:25:19.306186+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:19.306186+0000","last_undegraded":"2024-09-10T20:25:19.306186+0000","last_fullsized":"2024-09-10T20:25:19.306186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37396480,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7588","reported_seq":7755,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:18.452395+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:18.452395+0000","last_peered":"2024-09-10T20:25:18.452395+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:18.452395+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7588,"log_dups_size":0,"ondisk_log_size":7588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37288552,"num_objects":4553,"num_object_clones":0,"num_object_copies":18212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7587,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7755","reported_seq":11924,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:19.266350+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:19.266350+0000","last_peered":"2024-09-10T20:25:19.266350+0000","last_clean":"2024-09-10T20:25:19.266350+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:19.266350+0000","last_undegraded":"2024-09-10T20:25:19.266350+0000","last_fullsized":"2024-09-10T20:25:19.266350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7648","reported_seq":7929,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.428275+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:21.428275+0000","last_peered":"2024-09-10T20:25:21.428275+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:21.428275+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37033120,"num_objects":4521,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7627","reported_seq":7824,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.339834+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:21.339834+0000","last_peered":"2024-09-10T20:25:21.339834+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:21.339834+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36626432,"num_objects":4471,"num_object_clones":0,"num_object_copies":17884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7597","reported_seq":7777,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.632739+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:20.632739+0000","last_peered":"2024-09-10T20:25:20.632739+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:20.632739+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36639904,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2569}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7774","reported_seq":7919,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.534806+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:20.534806+0000","last_peered":"2024-09-10T20:25:20.534806+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:20.534806+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016736,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7772,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2508}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7585","reported_seq":11556,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:21.367467+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:21.367467+0000","last_peered":"2024-09-10T20:25:21.367467+0000","last_clean":"2024-09-10T20:25:21.367467+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:21.367467+0000","last_undegraded":"2024-09-10T20:25:21.367467+0000","last_fullsized":"2024-09-10T20:25:21.367467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36174416,"num_objects":4416,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7537","reported_seq":11507,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:18.314238+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:18.314238+0000","last_peered":"2024-09-10T20:25:18.314238+0000","last_clean":"2024-09-10T20:25:18.314238+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:18.314238+0000","last_undegraded":"2024-09-10T20:25:18.314238+0000","last_fullsized":"2024-09-10T20:25:18.314238+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37724160,"num_objects":4605,"num_object_clones":0,"num_object_copies":18420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":175,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7616","reported_seq":7866,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.314010+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:21.314010+0000","last_peered":"2024-09-10T20:25:21.314010+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:21.314010+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7616,"log_dups_size":0,"ondisk_log_size":7616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36835056,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7659","reported_seq":7926,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.011235+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:21.011235+0000","last_peered":"2024-09-10T20:25:21.011235+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:21.011235+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7659,"log_dups_size":0,"ondisk_log_size":7659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36911696,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7658,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2631}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7429","reported_seq":7587,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:18.334556+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:18.334556+0000","last_peered":"2024-09-10T20:25:18.334556+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:18.334556+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36059728,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7673","reported_seq":7917,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.396862+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:21.396862+0000","last_peered":"2024-09-10T20:25:21.396862+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:21.396862+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213920,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7752","reported_seq":7914,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:18.201869+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:18.201869+0000","last_peered":"2024-09-10T20:25:18.201869+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:18.201869+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7752,"log_dups_size":0,"ondisk_log_size":7752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38474912,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7750,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2768}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7649","reported_seq":8281,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.716125+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:20.716125+0000","last_peered":"2024-09-10T20:25:20.716125+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:20.716125+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36664480,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":591852552,"num_objects":72252,"num_object_clones":0,"num_object_copies":289008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102093,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1026,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1164357632,"data_stored":1164357632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122219,"ondisk_log_size":122219,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117067,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721092,"kb_used_data":288176,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93650748,"statfs":{"total":96636764160,"available":95898365952,"internally_reserved":0,"allocated":295092224,"data_stored":294879163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,4,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723952,"kb_used_data":291552,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93647888,"statfs":{"total":96636764160,"available":95895437312,"internally_reserved":0,"allocated":298549248,"data_stored":298336187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580880,"kb_used_data":270144,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93790960,"statfs":{"total":96636764160,"available":96041943040,"internally_reserved":0,"allocated":276627456,"data_stored":276414395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722632,"kb_used_data":289764,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93649208,"statfs":{"total":96636764160,"available":95896788992,"internally_reserved":0,"allocated":296718336,"data_stored":296505275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,2,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295477248,"data_stored":295477248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276279296,"data_stored":276279296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":297857024,"data_stored":297857024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294744064,"data_stored":294744064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:22.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:23.212 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:23.213 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:23.234 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-10T20:25:21.998509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591852552,"num_objects":72252,"num_object_clones":0,"num_object_copies":289008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102093,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1026,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122219,"ondisk_log_size":122219,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748556,"kb_used_data":1139636,"kb_used_omap":14,"kb_used_meta":1608817,"kb_avail":374738804,"statfs":{"total":386547056640,"available":383732535296,"internally_reserved":0,"allocated":1166987264,"data_stored":1166135020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1647429174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,10,13,11,2,0,2,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27451392,"num_objects":-3351,"num_object_clones":0,"num_object_copies":-13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004723"},"pg_stats":[{"pgid":"1.f","version":"40'7643","reported_seq":7890,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.310636+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:21.310636+0000","last_peered":"2024-09-10T20:25:21.310636+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:21.310636+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7643,"log_dups_size":0,"ondisk_log_size":7643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36691968,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7643,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7687","reported_seq":7818,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:19.306186+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:19.306186+0000","last_peered":"2024-09-10T20:25:19.306186+0000","last_clean":"2024-09-10T20:25:19.306186+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:19.306186+0000","last_undegraded":"2024-09-10T20:25:19.306186+0000","last_fullsized":"2024-09-10T20:25:19.306186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37396480,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7588","reported_seq":7755,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:18.452395+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:18.452395+0000","last_peered":"2024-09-10T20:25:18.452395+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:18.452395+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7588,"log_dups_size":0,"ondisk_log_size":7588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37288552,"num_objects":4553,"num_object_clones":0,"num_object_copies":18212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7587,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7755","reported_seq":11924,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:19.266350+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:19.266350+0000","last_peered":"2024-09-10T20:25:19.266350+0000","last_clean":"2024-09-10T20:25:19.266350+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:19.266350+0000","last_undegraded":"2024-09-10T20:25:19.266350+0000","last_fullsized":"2024-09-10T20:25:19.266350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7648","reported_seq":7929,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.428275+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:21.428275+0000","last_peered":"2024-09-10T20:25:21.428275+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:21.428275+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37033120,"num_objects":4521,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7627","reported_seq":7824,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.339834+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:21.339834+0000","last_peered":"2024-09-10T20:25:21.339834+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:21.339834+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36626432,"num_objects":4471,"num_object_clones":0,"num_object_copies":17884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7597","reported_seq":7777,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.632739+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:20.632739+0000","last_peered":"2024-09-10T20:25:20.632739+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:20.632739+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36639904,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2569}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7774","reported_seq":7919,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.534806+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:20.534806+0000","last_peered":"2024-09-10T20:25:20.534806+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:20.534806+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016736,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7772,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2508}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7585","reported_seq":11556,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:21.367467+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:21.367467+0000","last_peered":"2024-09-10T20:25:21.367467+0000","last_clean":"2024-09-10T20:25:21.367467+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:21.367467+0000","last_undegraded":"2024-09-10T20:25:21.367467+0000","last_fullsized":"2024-09-10T20:25:21.367467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36174416,"num_objects":4416,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7537","reported_seq":11507,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:18.314238+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:18.314238+0000","last_peered":"2024-09-10T20:25:18.314238+0000","last_clean":"2024-09-10T20:25:18.314238+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:18.314238+0000","last_undegraded":"2024-09-10T20:25:18.314238+0000","last_fullsized":"2024-09-10T20:25:18.314238+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37724160,"num_objects":4605,"num_object_clones":0,"num_object_copies":18420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":175,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7616","reported_seq":7866,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.314010+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:21.314010+0000","last_peered":"2024-09-10T20:25:21.314010+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:21.314010+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7616,"log_dups_size":0,"ondisk_log_size":7616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36835056,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7659","reported_seq":7926,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.011235+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:21.011235+0000","last_peered":"2024-09-10T20:25:21.011235+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:21.011235+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7659,"log_dups_size":0,"ondisk_log_size":7659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36911696,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7658,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2631}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7429","reported_seq":7587,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:18.334556+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:18.334556+0000","last_peered":"2024-09-10T20:25:18.334556+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:18.334556+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36059728,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7673","reported_seq":7917,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.396862+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:21.396862+0000","last_peered":"2024-09-10T20:25:21.396862+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:21.396862+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213920,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7752","reported_seq":7914,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:18.201869+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:18.201869+0000","last_peered":"2024-09-10T20:25:18.201869+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:18.201869+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7752,"log_dups_size":0,"ondisk_log_size":7752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38474912,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7750,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2768}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7649","reported_seq":8281,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.716125+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:20.716125+0000","last_peered":"2024-09-10T20:25:20.716125+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:20.716125+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36664480,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":591852552,"num_objects":72252,"num_object_clones":0,"num_object_copies":289008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102093,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1026,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1164357632,"data_stored":1164357632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122219,"ondisk_log_size":122219,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117067,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721092,"kb_used_data":288176,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93650748,"statfs":{"total":96636764160,"available":95898365952,"internally_reserved":0,"allocated":295092224,"data_stored":294879163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,4,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723952,"kb_used_data":291552,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93647888,"statfs":{"total":96636764160,"available":95895437312,"internally_reserved":0,"allocated":298549248,"data_stored":298336187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580880,"kb_used_data":270144,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93790960,"statfs":{"total":96636764160,"available":96041943040,"internally_reserved":0,"allocated":276627456,"data_stored":276414395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722632,"kb_used_data":289764,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93649208,"statfs":{"total":96636764160,"available":95896788992,"internally_reserved":0,"allocated":296718336,"data_stored":296505275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,2,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295477248,"data_stored":295477248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276279296,"data_stored":276279296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":297857024,"data_stored":297857024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294744064,"data_stored":294744064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:23.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:23.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'7643", 'reported_seq': 7890, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:21.310636+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:21.310636+0000', 'last_peered': '2024-09-10T20:25:21.310636+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:21.310636+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7643, 'log_dups_size': 0, 'ondisk_log_size': 7643, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36691968, 'num_objects': 4479, 'num_object_clones': 0, 'num_object_copies': 17916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1872, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7643, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2607}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:23.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:23.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7588", 'reported_seq': 7755, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:18.452395+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:18.452395+0000', 'last_peered': '2024-09-10T20:25:18.452395+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:18.452395+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7588, 'log_dups_size': 0, 'ondisk_log_size': 7588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37288552, 'num_objects': 4553, 'num_object_clones': 0, 'num_object_copies': 18212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1926, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4553, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7587, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2628}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:23.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:23.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'7648", 'reported_seq': 7929, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:21.428275+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:21.428275+0000', 'last_peered': '2024-09-10T20:25:21.428275+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:21.428275+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7648, 'log_dups_size': 0, 'ondisk_log_size': 7648, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37033120, 'num_objects': 4521, 'num_object_clones': 0, 'num_object_copies': 18084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1876, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7646, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 108, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7627", 'reported_seq': 7824, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:21.339834+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:21.339834+0000', 'last_peered': '2024-09-10T20:25:21.339834+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:21.339834+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7627, 'log_dups_size': 0, 'ondisk_log_size': 7627, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36626432, 'num_objects': 4471, 'num_object_clones': 0, 'num_object_copies': 17884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1869, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7627, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2602}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7597", 'reported_seq': 7777, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:20.632739+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:20.632739+0000', 'last_peered': '2024-09-10T20:25:20.632739+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:20.632739+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7597, 'log_dups_size': 0, 'ondisk_log_size': 7597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36639904, 'num_objects': 4473, 'num_object_clones': 0, 'num_object_copies': 17892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1906, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7595, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2569}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'7774", 'reported_seq': 7919, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:20.534806+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:20.534806+0000', 'last_peered': '2024-09-10T20:25:20.534806+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:20.534806+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7774, 'log_dups_size': 0, 'ondisk_log_size': 7774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37016736, 'num_objects': 4519, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7772, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2508}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7616", 'reported_seq': 7866, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:21.314010+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:21.314010+0000', 'last_peered': '2024-09-10T20:25:21.314010+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:21.314010+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7616, 'log_dups_size': 0, 'ondisk_log_size': 7616, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36835056, 'num_objects': 4497, 'num_object_clones': 0, 'num_object_copies': 17988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1870, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7613, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 92, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2630}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:23.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'7659", 'reported_seq': 7926, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:21.011235+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:21.011235+0000', 'last_peered': '2024-09-10T20:25:21.011235+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:21.011235+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7659, 'log_dups_size': 0, 'ondisk_log_size': 7659, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36911696, 'num_objects': 4506, 'num_object_clones': 0, 'num_object_copies': 18024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1876, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7658, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 104, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7429", 'reported_seq': 7587, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:18.334556+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:18.334556+0000', 'last_peered': '2024-09-10T20:25:18.334556+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:18.334556+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7429, 'log_dups_size': 0, 'ondisk_log_size': 7429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36059728, 'num_objects': 4402, 'num_object_clones': 0, 'num_object_copies': 17608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7428, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'7673", 'reported_seq': 7917, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:21.396862+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:21.396862+0000', 'last_peered': '2024-09-10T20:25:21.396862+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:21.396862+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7673, 'log_dups_size': 0, 'ondisk_log_size': 7673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36213920, 'num_objects': 4421, 'num_object_clones': 0, 'num_object_copies': 17684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1889, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7671, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'7752", 'reported_seq': 7914, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:18.201869+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:18.201869+0000', 'last_peered': '2024-09-10T20:25:18.201869+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:18.201869+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7752, 'log_dups_size': 0, 'ondisk_log_size': 7752, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38474912, 'num_objects': 4697, 'num_object_clones': 0, 'num_object_copies': 18788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7750, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2768}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7649", 'reported_seq': 8281, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:20.716125+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:20.716125+0000', 'last_peered': '2024-09-10T20:25:20.716125+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:20.716125+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7649, 'log_dups_size': 0, 'ondisk_log_size': 7649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36664480, 'num_objects': 4476, 'num_object_clones': 0, 'num_object_copies': 17904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1706, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7647, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 277, 'num_bytes_recovered': 2023424, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2770}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:23.239 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:23.993 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22650/289008 objects degraded (7.837%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":412,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72252,"data_bytes":591852552,"bytes_used":2814521344,"bytes_avail":383732535296,"bytes_total":386547056640,"degraded_objects":22650,"degraded_total":289008,"degraded_ratio":0.078371532967945523,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":105771,"recovering_keys_per_sec":0,"num_objects_recovered":279,"num_bytes_recovered":1269760,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":279},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:24.019 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:24.019 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:24.626 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:24.626 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:24.654 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-10T20:25:23.999361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589362184,"num_objects":71948,"num_object_clones":0,"num_object_copies":287792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102397,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122523,"ondisk_log_size":122523,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751936,"kb_used_data":1134684,"kb_used_omap":14,"kb_used_meta":1617137,"kb_avail":374735424,"statfs":{"total":386547056640,"available":383729074176,"internally_reserved":0,"allocated":1161916416,"data_stored":1161064172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1655948854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,2,4,7,10,11,2,0,2,1,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23437312,"num_objects":-2861,"num_object_clones":0,"num_object_copies":-11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2861,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005080"},"pg_stats":[{"pgid":"1.f","version":"40'7643","reported_seq":7890,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.310636+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:21.310636+0000","last_peered":"2024-09-10T20:25:21.310636+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:21.310636+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7643,"log_dups_size":0,"ondisk_log_size":7643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36691968,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7643,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7687","reported_seq":7818,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:19.306186+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:19.306186+0000","last_peered":"2024-09-10T20:25:19.306186+0000","last_clean":"2024-09-10T20:25:19.306186+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:19.306186+0000","last_undegraded":"2024-09-10T20:25:19.306186+0000","last_fullsized":"2024-09-10T20:25:19.306186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37396480,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7670","reported_seq":7844,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.474665+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:23.474665+0000","last_peered":"2024-09-10T20:25:23.474665+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:23.474665+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36616808,"num_objects":4471,"num_object_clones":0,"num_object_copies":17884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7669,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7755","reported_seq":11924,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:19.266350+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:19.266350+0000","last_peered":"2024-09-10T20:25:19.266350+0000","last_clean":"2024-09-10T20:25:19.266350+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:19.266350+0000","last_undegraded":"2024-09-10T20:25:19.266350+0000","last_fullsized":"2024-09-10T20:25:19.266350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7648","reported_seq":7929,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.428275+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:21.428275+0000","last_peered":"2024-09-10T20:25:21.428275+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:21.428275+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37033120,"num_objects":4521,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7627","reported_seq":7824,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.339834+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:21.339834+0000","last_peered":"2024-09-10T20:25:21.339834+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:21.339834+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36626432,"num_objects":4471,"num_object_clones":0,"num_object_copies":17884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7597","reported_seq":7777,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.632739+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:20.632739+0000","last_peered":"2024-09-10T20:25:20.632739+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:20.632739+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36639904,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2569}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7774","reported_seq":7919,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.534806+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:20.534806+0000","last_peered":"2024-09-10T20:25:20.534806+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:20.534806+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016736,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7772,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2508}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7585","reported_seq":11556,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:21.367467+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:21.367467+0000","last_peered":"2024-09-10T20:25:21.367467+0000","last_clean":"2024-09-10T20:25:21.367467+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:21.367467+0000","last_undegraded":"2024-09-10T20:25:21.367467+0000","last_fullsized":"2024-09-10T20:25:21.367467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36174416,"num_objects":4416,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7605","reported_seq":11575,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:23.351852+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:23.351852+0000","last_peered":"2024-09-10T20:25:23.351852+0000","last_clean":"2024-09-10T20:25:23.351852+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:23.351852+0000","last_undegraded":"2024-09-10T20:25:23.351852+0000","last_fullsized":"2024-09-10T20:25:23.351852+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7605,"log_dups_size":0,"ondisk_log_size":7605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167104,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":243,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7616","reported_seq":7866,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.314010+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:21.314010+0000","last_peered":"2024-09-10T20:25:21.314010+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:21.314010+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7616,"log_dups_size":0,"ondisk_log_size":7616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36835056,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7659","reported_seq":7926,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.011235+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:21.011235+0000","last_peered":"2024-09-10T20:25:21.011235+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:21.011235+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7659,"log_dups_size":0,"ondisk_log_size":7659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36911696,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7658,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2631}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7501","reported_seq":7659,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.444611+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:23.444611+0000","last_peered":"2024-09-10T20:25:23.444611+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:23.444611+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35469904,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7500,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7673","reported_seq":7917,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:21.396862+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:21.396862+0000","last_peered":"2024-09-10T20:25:21.396862+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:21.396862+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213920,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7834","reported_seq":7997,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.366713+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:23.366713+0000","last_peered":"2024-09-10T20:25:23.366713+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:23.366713+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37803168,"num_objects":4615,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7832,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7649","reported_seq":8281,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:20.716125+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:20.716125+0000","last_peered":"2024-09-10T20:25:20.716125+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:20.716125+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36664480,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":589362184,"num_objects":71948,"num_object_clones":0,"num_object_copies":287792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102397,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1159229440,"data_stored":1159229440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122523,"ondisk_log_size":122523,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117067,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721092,"kb_used_data":288176,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93650748,"statfs":{"total":96636764160,"available":95898365952,"internally_reserved":0,"allocated":295092224,"data_stored":294879163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,4,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727332,"kb_used_data":286600,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93644508,"statfs":{"total":96636764160,"available":95891976192,"internally_reserved":0,"allocated":293478400,"data_stored":293265339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,1,0,2,1,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580880,"kb_used_data":270144,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93790960,"statfs":{"total":96636764160,"available":96041943040,"internally_reserved":0,"allocated":276627456,"data_stored":276414395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722632,"kb_used_data":289764,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93649208,"statfs":{"total":96636764160,"available":95896788992,"internally_reserved":0,"allocated":296718336,"data_stored":296505275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,2,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295477248,"data_stored":295477248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276279296,"data_stored":276279296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292728832,"data_stored":292728832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294744064,"data_stored":294744064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:27.656 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:28.242 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:28.242 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:28.268 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-10T20:25:28.002590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581948424,"num_objects":71043,"num_object_clones":0,"num_object_copies":284172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103302,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1101,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123428,"ondisk_log_size":123428,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745200,"kb_used_data":1123800,"kb_used_omap":14,"kb_used_meta":1621297,"kb_avail":374742160,"statfs":{"total":386547056640,"available":383735971840,"internally_reserved":0,"allocated":1150771200,"data_stored":1149918956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1660208694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,16,9,4,10,8,9,8,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24969216,"num_objects":-3048,"num_object_clones":0,"num_object_copies":-12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006649"},"pg_stats":[{"pgid":"1.f","version":"40'7720","reported_seq":7972,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.415467+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:26.415467+0000","last_peered":"2024-09-10T20:25:26.415467+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:26.415467+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7764","reported_seq":7895,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:24.305102+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:24.305102+0000","last_peered":"2024-09-10T20:25:24.305102+0000","last_clean":"2024-09-10T20:25:24.305102+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:24.305102+0000","last_undegraded":"2024-09-10T20:25:24.305102+0000","last_fullsized":"2024-09-10T20:25:24.305102+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36765696,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4205,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7670","reported_seq":7844,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.474665+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:23.474665+0000","last_peered":"2024-09-10T20:25:23.474665+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:23.474665+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36616808,"num_objects":4471,"num_object_clones":0,"num_object_copies":17884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7669,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7834","reported_seq":12003,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:24.275719+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:24.275719+0000","last_peered":"2024-09-10T20:25:24.275719+0000","last_clean":"2024-09-10T20:25:24.275719+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:24.275719+0000","last_undegraded":"2024-09-10T20:25:24.275719+0000","last_fullsized":"2024-09-10T20:25:24.275719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37453824,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2094,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7719","reported_seq":8006,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.390472+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:26.390472+0000","last_peered":"2024-09-10T20:25:26.390472+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:26.390472+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36451488,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7707","reported_seq":7909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.467817+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:26.467817+0000","last_peered":"2024-09-10T20:25:26.467817+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:26.467817+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35971072,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7707,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7664","reported_seq":7846,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.541489+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:25.541489+0000","last_peered":"2024-09-10T20:25:25.541489+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:25.541489+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7664,"log_dups_size":0,"ondisk_log_size":7664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36091040,"num_objects":4406,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7662,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7848","reported_seq":7995,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.764611+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:25.764611+0000","last_peered":"2024-09-10T20:25:25.764611+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:25.764611+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36410528,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7656","reported_seq":11627,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:26.355673+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:26.355673+0000","last_peered":"2024-09-10T20:25:26.355673+0000","last_clean":"2024-09-10T20:25:26.355673+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:26.355673+0000","last_undegraded":"2024-09-10T20:25:26.355673+0000","last_fullsized":"2024-09-10T20:25:26.355673+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7656,"log_dups_size":0,"ondisk_log_size":7656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35592784,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7605","reported_seq":11575,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:23.351852+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:23.351852+0000","last_peered":"2024-09-10T20:25:23.351852+0000","last_clean":"2024-09-10T20:25:23.351852+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:23.351852+0000","last_undegraded":"2024-09-10T20:25:23.351852+0000","last_fullsized":"2024-09-10T20:25:23.351852+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7605,"log_dups_size":0,"ondisk_log_size":7605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167104,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":243,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7696","reported_seq":7952,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.327776+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:26.327776+0000","last_peered":"2024-09-10T20:25:26.327776+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:26.327776+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7696,"log_dups_size":0,"ondisk_log_size":7696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36179696,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7734","reported_seq":8009,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.186619+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:26.186619+0000","last_peered":"2024-09-10T20:25:26.186619+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:26.186619+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36297296,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2560}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7501","reported_seq":7659,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.444611+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:23.444611+0000","last_peered":"2024-09-10T20:25:23.444611+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:23.444611+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35469904,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7500,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7743","reported_seq":7988,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.471912+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:26.471912+0000","last_peered":"2024-09-10T20:25:26.471912+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:26.471912+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7743,"log_dups_size":0,"ondisk_log_size":7743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35640480,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2465}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7834","reported_seq":7997,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.366713+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:23.366713+0000","last_peered":"2024-09-10T20:25:23.366713+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:23.366713+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37803168,"num_objects":4615,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7832,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7733","reported_seq":8467,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.772237+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:25.772237+0000","last_peered":"2024-09-10T20:25:25.772237+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:25.772237+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35976352,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":581948424,"num_objects":71043,"num_object_clones":0,"num_object_copies":284172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103302,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1101,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147813888,"data_stored":1147813888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123428,"ondisk_log_size":123428,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117068,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724556,"kb_used_data":283320,"kb_used_omap":4,"kb_used_meta":441211,"kb_avail":93647284,"statfs":{"total":96636764160,"available":95894818816,"internally_reserved":0,"allocated":290119680,"data_stored":289906619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":451800784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,5,1,4,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727332,"kb_used_data":286600,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93644508,"statfs":{"total":96636764160,"available":95891976192,"internally_reserved":0,"allocated":293478400,"data_stored":293265339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,1,0,2,1,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571436,"kb_used_data":269012,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93800404,"statfs":{"total":96636764160,"available":96051613696,"internally_reserved":0,"allocated":275468288,"data_stored":275255227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721876,"kb_used_data":284868,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93649964,"statfs":{"total":96636764160,"available":95897563136,"internally_reserved":0,"allocated":291704832,"data_stored":291491771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,4,1,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290369536,"data_stored":290369536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275107840,"data_stored":275107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292728832,"data_stored":292728832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289607680,"data_stored":289607680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:28.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:28.893 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:28.893 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:28.912 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-10T20:25:28.002590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581948424,"num_objects":71043,"num_object_clones":0,"num_object_copies":284172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103302,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1101,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123428,"ondisk_log_size":123428,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745200,"kb_used_data":1123800,"kb_used_omap":14,"kb_used_meta":1621297,"kb_avail":374742160,"statfs":{"total":386547056640,"available":383735971840,"internally_reserved":0,"allocated":1150771200,"data_stored":1149918956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1660208694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,16,9,4,10,8,9,8,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24969216,"num_objects":-3048,"num_object_clones":0,"num_object_copies":-12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006649"},"pg_stats":[{"pgid":"1.f","version":"40'7720","reported_seq":7972,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.415467+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:26.415467+0000","last_peered":"2024-09-10T20:25:26.415467+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:26.415467+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7764","reported_seq":7895,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:24.305102+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:24.305102+0000","last_peered":"2024-09-10T20:25:24.305102+0000","last_clean":"2024-09-10T20:25:24.305102+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:24.305102+0000","last_undegraded":"2024-09-10T20:25:24.305102+0000","last_fullsized":"2024-09-10T20:25:24.305102+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36765696,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4205,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7670","reported_seq":7844,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.474665+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:23.474665+0000","last_peered":"2024-09-10T20:25:23.474665+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:23.474665+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36616808,"num_objects":4471,"num_object_clones":0,"num_object_copies":17884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7669,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7834","reported_seq":12003,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:24.275719+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:24.275719+0000","last_peered":"2024-09-10T20:25:24.275719+0000","last_clean":"2024-09-10T20:25:24.275719+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:24.275719+0000","last_undegraded":"2024-09-10T20:25:24.275719+0000","last_fullsized":"2024-09-10T20:25:24.275719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37453824,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2094,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7719","reported_seq":8006,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.390472+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:26.390472+0000","last_peered":"2024-09-10T20:25:26.390472+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:26.390472+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36451488,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7707","reported_seq":7909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.467817+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:26.467817+0000","last_peered":"2024-09-10T20:25:26.467817+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:26.467817+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35971072,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7707,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7664","reported_seq":7846,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.541489+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:25.541489+0000","last_peered":"2024-09-10T20:25:25.541489+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:25.541489+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7664,"log_dups_size":0,"ondisk_log_size":7664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36091040,"num_objects":4406,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7662,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7848","reported_seq":7995,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.764611+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:25.764611+0000","last_peered":"2024-09-10T20:25:25.764611+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:25.764611+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36410528,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7656","reported_seq":11627,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:26.355673+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:26.355673+0000","last_peered":"2024-09-10T20:25:26.355673+0000","last_clean":"2024-09-10T20:25:26.355673+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:26.355673+0000","last_undegraded":"2024-09-10T20:25:26.355673+0000","last_fullsized":"2024-09-10T20:25:26.355673+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7656,"log_dups_size":0,"ondisk_log_size":7656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35592784,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7605","reported_seq":11575,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:23.351852+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:23.351852+0000","last_peered":"2024-09-10T20:25:23.351852+0000","last_clean":"2024-09-10T20:25:23.351852+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:23.351852+0000","last_undegraded":"2024-09-10T20:25:23.351852+0000","last_fullsized":"2024-09-10T20:25:23.351852+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7605,"log_dups_size":0,"ondisk_log_size":7605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167104,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":243,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7696","reported_seq":7952,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.327776+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:26.327776+0000","last_peered":"2024-09-10T20:25:26.327776+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:26.327776+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7696,"log_dups_size":0,"ondisk_log_size":7696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36179696,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7734","reported_seq":8009,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.186619+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:26.186619+0000","last_peered":"2024-09-10T20:25:26.186619+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:26.186619+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36297296,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2560}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7501","reported_seq":7659,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.444611+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:23.444611+0000","last_peered":"2024-09-10T20:25:23.444611+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:23.444611+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35469904,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7500,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7743","reported_seq":7988,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.471912+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:26.471912+0000","last_peered":"2024-09-10T20:25:26.471912+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:26.471912+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7743,"log_dups_size":0,"ondisk_log_size":7743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35640480,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2465}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7834","reported_seq":7997,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:23.366713+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:23.366713+0000","last_peered":"2024-09-10T20:25:23.366713+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:23.366713+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37803168,"num_objects":4615,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7832,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7733","reported_seq":8467,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.772237+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:25.772237+0000","last_peered":"2024-09-10T20:25:25.772237+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:25.772237+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35976352,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":581948424,"num_objects":71043,"num_object_clones":0,"num_object_copies":284172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103302,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1101,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147813888,"data_stored":1147813888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123428,"ondisk_log_size":123428,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117068,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724556,"kb_used_data":283320,"kb_used_omap":4,"kb_used_meta":441211,"kb_avail":93647284,"statfs":{"total":96636764160,"available":95894818816,"internally_reserved":0,"allocated":290119680,"data_stored":289906619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":451800784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,5,1,4,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727332,"kb_used_data":286600,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93644508,"statfs":{"total":96636764160,"available":95891976192,"internally_reserved":0,"allocated":293478400,"data_stored":293265339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,1,0,2,1,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571436,"kb_used_data":269012,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93800404,"statfs":{"total":96636764160,"available":96051613696,"internally_reserved":0,"allocated":275468288,"data_stored":275255227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721876,"kb_used_data":284868,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93649964,"statfs":{"total":96636764160,"available":95897563136,"internally_reserved":0,"allocated":291704832,"data_stored":291491771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,4,1,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290369536,"data_stored":290369536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275107840,"data_stored":275107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292728832,"data_stored":292728832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289607680,"data_stored":289607680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:28.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:28.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'7720", 'reported_seq': 7972, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:26.415467+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:26.415467+0000', 'last_peered': '2024-09-10T20:25:26.415467+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:26.415467+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7720, 'log_dups_size': 0, 'ondisk_log_size': 7720, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36061184, 'num_objects': 4402, 'num_object_clones': 0, 'num_object_copies': 17608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1869, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7720, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 97, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2533}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:28.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:28.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7670", 'reported_seq': 7844, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:23.474665+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:23.474665+0000', 'last_peered': '2024-09-10T20:25:23.474665+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:23.474665+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7670, 'log_dups_size': 0, 'ondisk_log_size': 7670, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36616808, 'num_objects': 4471, 'num_object_clones': 0, 'num_object_copies': 17884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1923, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4471, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7669, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2549}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'7719", 'reported_seq': 8006, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:26.390472+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:26.390472+0000', 'last_peered': '2024-09-10T20:25:26.390472+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:26.390472+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7719, 'log_dups_size': 0, 'ondisk_log_size': 7719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36451488, 'num_objects': 4450, 'num_object_clones': 0, 'num_object_copies': 17800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1873, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7717, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2579}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7707", 'reported_seq': 7909, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:26.467817+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:26.467817+0000', 'last_peered': '2024-09-10T20:25:26.467817+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:26.467817+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7707, 'log_dups_size': 0, 'ondisk_log_size': 7707, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35971072, 'num_objects': 4391, 'num_object_clones': 0, 'num_object_copies': 17564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1866, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7707, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 72, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2525}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7664", 'reported_seq': 7846, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:25.541489+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:25.541489+0000', 'last_peered': '2024-09-10T20:25:25.541489+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:25.541489+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7664, 'log_dups_size': 0, 'ondisk_log_size': 7664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36091040, 'num_objects': 4406, 'num_object_clones': 0, 'num_object_copies': 17624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7662, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2504}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'7848", 'reported_seq': 7995, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:25.764611+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:25.764611+0000', 'last_peered': '2024-09-10T20:25:25.764611+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:25.764611+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7848, 'log_dups_size': 0, 'ondisk_log_size': 7848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36410528, 'num_objects': 4445, 'num_object_clones': 0, 'num_object_copies': 17780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2012, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7846, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2435}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7696", 'reported_seq': 7952, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:26.327776+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:26.327776+0000', 'last_peered': '2024-09-10T20:25:26.327776+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:26.327776+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7696, 'log_dups_size': 0, 'ondisk_log_size': 7696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36179696, 'num_objects': 4417, 'num_object_clones': 0, 'num_object_copies': 17668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1867, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7693, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 95, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2553}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:28.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'7734", 'reported_seq': 8009, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:26.186619+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:26.186619+0000', 'last_peered': '2024-09-10T20:25:26.186619+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:26.186619+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7734, 'log_dups_size': 0, 'ondisk_log_size': 7734, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36297296, 'num_objects': 4431, 'num_object_clones': 0, 'num_object_copies': 17724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1872, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7733, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 108, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2560}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7501", 'reported_seq': 7659, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:23.444611+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:23.444611+0000', 'last_peered': '2024-09-10T20:25:23.444611+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:23.444611+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7501, 'log_dups_size': 0, 'ondisk_log_size': 7501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35469904, 'num_objects': 4330, 'num_object_clones': 0, 'num_object_copies': 17320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7500, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'7743", 'reported_seq': 7988, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:26.471912+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:26.471912+0000', 'last_peered': '2024-09-10T20:25:26.471912+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:26.471912+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7743, 'log_dups_size': 0, 'ondisk_log_size': 7743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35640480, 'num_objects': 4351, 'num_object_clones': 0, 'num_object_copies': 17404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1888, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7741, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'7834", 'reported_seq': 7997, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:23.366713+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:23.366713+0000', 'last_peered': '2024-09-10T20:25:23.366713+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:23.366713+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7834, 'log_dups_size': 0, 'ondisk_log_size': 7834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37803168, 'num_objects': 4615, 'num_object_clones': 0, 'num_object_copies': 18460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1930, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7832, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:28.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:28.916 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7733", 'reported_seq': 8467, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:25.772237+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:25.772237+0000', 'last_peered': '2024-09-10T20:25:25.772237+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:25.772237+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7733, 'log_dups_size': 0, 'ondisk_log_size': 7733, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35976352, 'num_objects': 4392, 'num_object_clones': 0, 'num_object_copies': 17568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1655, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7731, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 328, 'num_bytes_recovered': 2433024, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:28.916 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:29.663 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:29.663 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22575/284172 objects degraded (7.944%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":418,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71043,"data_bytes":581948424,"bytes_used":2811084800,"bytes_avail":383735971840,"bytes_total":386547056640,"degraded_objects":22575,"degraded_total":284172,"degraded_ratio":0.079441324268400823,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":69593,"recovering_keys_per_sec":0,"num_objects_recovered":217,"num_bytes_recovered":835584,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:29.688 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:29.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:30.258 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:30.258 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:30.298 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-10T20:25:30.003235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577827848,"num_objects":70540,"num_object_clones":0,"num_object_copies":282160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103805,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123931,"ondisk_log_size":123931,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730664,"kb_used_data":1117604,"kb_used_omap":14,"kb_used_meta":1612977,"kb_avail":374756696,"statfs":{"total":386547056640,"available":383750856704,"internally_reserved":0,"allocated":1144426496,"data_stored":1143574252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1651689014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,11,9,11,11,6,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24141824,"num_objects":-2947,"num_object_clones":0,"num_object_copies":-11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006429"},"pg_stats":[{"pgid":"1.f","version":"40'7720","reported_seq":7972,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.415467+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:26.415467+0000","last_peered":"2024-09-10T20:25:26.415467+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:26.415467+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7847","reported_seq":7978,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.295786+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:29.295786+0000","last_peered":"2024-09-10T20:25:29.295786+0000","last_clean":"2024-09-10T20:25:29.295786+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:29.295786+0000","last_undegraded":"2024-09-10T20:25:29.295786+0000","last_fullsized":"2024-09-10T20:25:29.295786+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7847,"log_dups_size":0,"ondisk_log_size":7847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36085760,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7750","reported_seq":7929,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:28.447261+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:28.447261+0000","last_peered":"2024-09-10T20:25:28.447261+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:28.447261+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35961448,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7749,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7911","reported_seq":12080,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.390554+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:29.390554+0000","last_peered":"2024-09-10T20:25:29.390554+0000","last_clean":"2024-09-10T20:25:29.390554+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:29.390554+0000","last_undegraded":"2024-09-10T20:25:29.390554+0000","last_fullsized":"2024-09-10T20:25:29.390554+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36823040,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7719","reported_seq":8006,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.390472+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:26.390472+0000","last_peered":"2024-09-10T20:25:26.390472+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:26.390472+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36451488,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7707","reported_seq":7909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.467817+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:26.467817+0000","last_peered":"2024-09-10T20:25:26.467817+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:26.467817+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35971072,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7707,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7664","reported_seq":7846,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.541489+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:25.541489+0000","last_peered":"2024-09-10T20:25:25.541489+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:25.541489+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7664,"log_dups_size":0,"ondisk_log_size":7664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36091040,"num_objects":4406,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7662,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7848","reported_seq":7995,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.764611+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:25.764611+0000","last_peered":"2024-09-10T20:25:25.764611+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:25.764611+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36410528,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7656","reported_seq":11627,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:26.355673+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:26.355673+0000","last_peered":"2024-09-10T20:25:26.355673+0000","last_clean":"2024-09-10T20:25:26.355673+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:26.355673+0000","last_undegraded":"2024-09-10T20:25:26.355673+0000","last_fullsized":"2024-09-10T20:25:26.355673+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7656,"log_dups_size":0,"ondisk_log_size":7656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35592784,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7699","reported_seq":11669,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:28.354616+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:28.354616+0000","last_peered":"2024-09-10T20:25:28.354616+0000","last_clean":"2024-09-10T20:25:28.354616+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:28.354616+0000","last_undegraded":"2024-09-10T20:25:28.354616+0000","last_fullsized":"2024-09-10T20:25:28.354616+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7699,"log_dups_size":0,"ondisk_log_size":7699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36397056,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":337,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7696","reported_seq":7952,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.327776+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:26.327776+0000","last_peered":"2024-09-10T20:25:26.327776+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:26.327776+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7696,"log_dups_size":0,"ondisk_log_size":7696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36179696,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7734","reported_seq":8009,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.186619+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:26.186619+0000","last_peered":"2024-09-10T20:25:26.186619+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:26.186619+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36297296,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2560}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7589","reported_seq":7753,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:28.251075+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:28.251075+0000","last_peered":"2024-09-10T20:25:28.251075+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:28.251075+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34749008,"num_objects":4242,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7588,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2330}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7743","reported_seq":7988,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:26.471912+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:26.471912+0000","last_peered":"2024-09-10T20:25:26.471912+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:26.471912+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7743,"log_dups_size":0,"ondisk_log_size":7743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35640480,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2465}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7915","reported_seq":8083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:28.311121+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:28.311121+0000","last_peered":"2024-09-10T20:25:28.311121+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:28.311121+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7915,"log_dups_size":0,"ondisk_log_size":7915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37139616,"num_objects":4534,"num_object_clones":0,"num_object_copies":18136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7913,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2608}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7733","reported_seq":8467,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:25.772237+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:25.772237+0000","last_peered":"2024-09-10T20:25:25.772237+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:25.772237+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35976352,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":577827848,"num_objects":70540,"num_object_clones":0,"num_object_copies":282160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103805,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":2433024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141395456,"data_stored":1141395456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123931,"ondisk_log_size":123931,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117068,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724556,"kb_used_data":283320,"kb_used_omap":4,"kb_used_meta":441211,"kb_avail":93647284,"statfs":{"total":96636764160,"available":95894818816,"internally_reserved":0,"allocated":290119680,"data_stored":289906619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":451800784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,5,1,4,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709916,"kb_used_data":281676,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93661924,"statfs":{"total":96636764160,"available":95909810176,"internally_reserved":0,"allocated":288436224,"data_stored":288223163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574316,"kb_used_data":267740,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93797524,"statfs":{"total":96636764160,"available":96048664576,"internally_reserved":0,"allocated":274165760,"data_stored":273952699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721876,"kb_used_data":284868,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93649964,"statfs":{"total":96636764160,"available":95897563136,"internally_reserved":0,"allocated":291704832,"data_stored":291491771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,4,1,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290369536,"data_stored":290369536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273829888,"data_stored":273829888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287588352,"data_stored":287588352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289607680,"data_stored":289607680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:33.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:33.923 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:33.923 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:33.969 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-10T20:25:32.004077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":571634696,"num_objects":69784,"num_object_clones":0,"num_object_copies":279136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104561,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1176,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124687,"ondisk_log_size":124687,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699028,"kb_used_data":1106772,"kb_used_omap":14,"kb_used_meta":1592177,"kb_avail":374788332,"statfs":{"total":386547056640,"available":383783251968,"internally_reserved":0,"allocated":1133334528,"data_stored":1132482284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1630389814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,7,6,19,3,3,1,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26746880,"num_objects":-3265,"num_object_clones":0,"num_object_copies":-13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006446"},"pg_stats":[{"pgid":"1.f","version":"40'7790","reported_seq":8049,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.304292+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:31.304292+0000","last_peered":"2024-09-10T20:25:31.304292+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:31.304292+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35487744,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7790,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7847","reported_seq":7978,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.295786+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:29.295786+0000","last_peered":"2024-09-10T20:25:29.295786+0000","last_clean":"2024-09-10T20:25:29.295786+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:29.295786+0000","last_undegraded":"2024-09-10T20:25:29.295786+0000","last_fullsized":"2024-09-10T20:25:29.295786+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7847,"log_dups_size":0,"ondisk_log_size":7847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36085760,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7750","reported_seq":7929,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:28.447261+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:28.447261+0000","last_peered":"2024-09-10T20:25:28.447261+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:28.447261+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35961448,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7749,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7911","reported_seq":12080,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.390554+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:29.390554+0000","last_peered":"2024-09-10T20:25:29.390554+0000","last_clean":"2024-09-10T20:25:29.390554+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:29.390554+0000","last_undegraded":"2024-09-10T20:25:29.390554+0000","last_fullsized":"2024-09-10T20:25:29.390554+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36823040,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7787","reported_seq":8076,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.460564+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:31.460564+0000","last_peered":"2024-09-10T20:25:31.460564+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:31.460564+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35894432,"num_objects":4382,"num_object_clones":0,"num_object_copies":17528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7785,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7768","reported_seq":7975,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.103437+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:31.103437+0000","last_peered":"2024-09-10T20:25:31.103437+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:31.103437+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7768,"log_dups_size":0,"ondisk_log_size":7768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35471360,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7768,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7740","reported_seq":7924,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.792147+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:30.792147+0000","last_peered":"2024-09-10T20:25:30.792147+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:30.792147+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7740,"log_dups_size":0,"ondisk_log_size":7740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35468448,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2428}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7932","reported_seq":8080,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.805084+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:30.805084+0000","last_peered":"2024-09-10T20:25:30.805084+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:30.805084+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722400,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7719","reported_seq":11690,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:31.511812+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:31.511812+0000","last_peered":"2024-09-10T20:25:31.511812+0000","last_clean":"2024-09-10T20:25:31.511812+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:31.511812+0000","last_undegraded":"2024-09-10T20:25:31.511812+0000","last_fullsized":"2024-09-10T20:25:31.511812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35076688,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7699","reported_seq":11669,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:28.354616+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:28.354616+0000","last_peered":"2024-09-10T20:25:28.354616+0000","last_clean":"2024-09-10T20:25:28.354616+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:28.354616+0000","last_undegraded":"2024-09-10T20:25:28.354616+0000","last_fullsized":"2024-09-10T20:25:28.354616+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7699,"log_dups_size":0,"ondisk_log_size":7699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36397056,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":337,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7781","reported_seq":8039,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.457420+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:31.457420+0000","last_peered":"2024-09-10T20:25:31.457420+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:31.457420+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483376,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7825","reported_seq":8106,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.475375+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:31.475375+0000","last_peered":"2024-09-10T20:25:31.475375+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:31.475375+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7825,"log_dups_size":0,"ondisk_log_size":7825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35551824,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7589","reported_seq":7753,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:28.251075+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:28.251075+0000","last_peered":"2024-09-10T20:25:28.251075+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:28.251075+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34749008,"num_objects":4242,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7588,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2330}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7819","reported_seq":8073,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.400914+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:31.400914+0000","last_peered":"2024-09-10T20:25:31.400914+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:31.400914+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35017888,"num_objects":4275,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2393}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7915","reported_seq":8083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:28.311121+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:28.311121+0000","last_peered":"2024-09-10T20:25:28.311121+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:28.311121+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7915,"log_dups_size":0,"ondisk_log_size":7915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37139616,"num_objects":4534,"num_object_clones":0,"num_object_copies":18136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7913,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2608}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7815","reported_seq":8653,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.843281+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:30.843281+0000","last_peered":"2024-09-10T20:25:30.843281+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:30.843281+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7815,"log_dups_size":0,"ondisk_log_size":7815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304608,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7813,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2707}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":571634696,"num_objects":69784,"num_object_clones":0,"num_object_copies":279136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104561,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1176,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1131286528,"data_stored":1131286528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124687,"ondisk_log_size":124687,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707092,"kb_used_data":278340,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93664748,"statfs":{"total":96636764160,"available":95912701952,"internally_reserved":0,"allocated":285020160,"data_stored":284807099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709916,"kb_used_data":281676,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93661924,"statfs":{"total":96636764160,"available":95909810176,"internally_reserved":0,"allocated":288436224,"data_stored":288223163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574316,"kb_used_data":267740,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93797524,"statfs":{"total":96636764160,"available":96048664576,"internally_reserved":0,"allocated":274165760,"data_stored":273952699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707704,"kb_used_data":279016,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93664136,"statfs":{"total":96636764160,"available":95912075264,"internally_reserved":0,"allocated":285712384,"data_stored":285499323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285282304,"data_stored":285282304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273829888,"data_stored":273829888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287588352,"data_stored":287588352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284585984,"data_stored":284585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:33.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:34.571 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:34.571 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:34.597 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-10T20:25:34.004924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569013256,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104881,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125007,"ondisk_log_size":125007,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698488,"kb_used_data":1102072,"kb_used_omap":14,"kb_used_meta":1596337,"kb_avail":374788872,"statfs":{"total":386547056640,"available":383783804928,"internally_reserved":0,"allocated":1128521728,"data_stored":1127669484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1634649654},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,11,7,21,3,0,0,0,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22839296,"num_objects":-2788,"num_object_clones":0,"num_object_copies":-11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006414"},"pg_stats":[{"pgid":"1.f","version":"40'7790","reported_seq":8049,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.304292+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:31.304292+0000","last_peered":"2024-09-10T20:25:31.304292+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:31.304292+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35487744,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7790,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7847","reported_seq":7978,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.295786+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:29.295786+0000","last_peered":"2024-09-10T20:25:29.295786+0000","last_clean":"2024-09-10T20:25:29.295786+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:29.295786+0000","last_undegraded":"2024-09-10T20:25:29.295786+0000","last_fullsized":"2024-09-10T20:25:29.295786+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7847,"log_dups_size":0,"ondisk_log_size":7847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36085760,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7833","reported_seq":8023,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.492310+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:33.492310+0000","last_peered":"2024-09-10T20:25:33.492310+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:33.492310+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7833,"log_dups_size":0,"ondisk_log_size":7833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35281512,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7832,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2394}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7911","reported_seq":12080,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.390554+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:29.390554+0000","last_peered":"2024-09-10T20:25:29.390554+0000","last_clean":"2024-09-10T20:25:29.390554+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:29.390554+0000","last_undegraded":"2024-09-10T20:25:29.390554+0000","last_fullsized":"2024-09-10T20:25:29.390554+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36823040,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7787","reported_seq":8076,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.460564+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:31.460564+0000","last_peered":"2024-09-10T20:25:31.460564+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:31.460564+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35894432,"num_objects":4382,"num_object_clones":0,"num_object_copies":17528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7785,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7768","reported_seq":7975,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.103437+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:31.103437+0000","last_peered":"2024-09-10T20:25:31.103437+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:31.103437+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7768,"log_dups_size":0,"ondisk_log_size":7768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35471360,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7768,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7740","reported_seq":7924,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.792147+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:30.792147+0000","last_peered":"2024-09-10T20:25:30.792147+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:30.792147+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7740,"log_dups_size":0,"ondisk_log_size":7740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35468448,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2428}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7932","reported_seq":8080,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.805084+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:30.805084+0000","last_peered":"2024-09-10T20:25:30.805084+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:30.805084+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722400,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7719","reported_seq":11690,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:31.511812+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:31.511812+0000","last_peered":"2024-09-10T20:25:31.511812+0000","last_clean":"2024-09-10T20:25:31.511812+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:31.511812+0000","last_undegraded":"2024-09-10T20:25:31.511812+0000","last_fullsized":"2024-09-10T20:25:31.511812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35076688,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7785","reported_seq":11755,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:33.456185+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:33.456185+0000","last_peered":"2024-09-10T20:25:33.456185+0000","last_clean":"2024-09-10T20:25:33.456185+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:33.456185+0000","last_undegraded":"2024-09-10T20:25:33.456185+0000","last_fullsized":"2024-09-10T20:25:33.456185+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35692544,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7781","reported_seq":8039,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.457420+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:31.457420+0000","last_peered":"2024-09-10T20:25:31.457420+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:31.457420+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483376,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7825","reported_seq":8106,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.475375+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:31.475375+0000","last_peered":"2024-09-10T20:25:31.475375+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:31.475375+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7825,"log_dups_size":0,"ondisk_log_size":7825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35551824,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7667","reported_seq":7837,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.418897+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:33.418897+0000","last_peered":"2024-09-10T20:25:33.418897+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:33.418897+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34110032,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2255}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7819","reported_seq":8073,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.400914+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:31.400914+0000","last_peered":"2024-09-10T20:25:31.400914+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:31.400914+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35017888,"num_objects":4275,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2393}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7988","reported_seq":8164,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.261709+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:33.261709+0000","last_peered":"2024-09-10T20:25:33.261709+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:33.261709+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541600,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7815","reported_seq":8653,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.843281+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:30.843281+0000","last_peered":"2024-09-10T20:25:30.843281+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:30.843281+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7815,"log_dups_size":0,"ondisk_log_size":7815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304608,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7813,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2707}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569013256,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104881,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126354944,"data_stored":1126354944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125007,"ondisk_log_size":125007,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707092,"kb_used_data":278340,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93664748,"statfs":{"total":96636764160,"available":95912701952,"internally_reserved":0,"allocated":285020160,"data_stored":284807099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":276976,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":283623424,"data_stored":283410363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,5,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574316,"kb_used_data":267740,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93797524,"statfs":{"total":96636764160,"available":96048664576,"internally_reserved":0,"allocated":274165760,"data_stored":273952699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707704,"kb_used_data":279016,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93664136,"statfs":{"total":96636764160,"available":95912075264,"internally_reserved":0,"allocated":285712384,"data_stored":285499323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285282304,"data_stored":285282304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273829888,"data_stored":273829888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282656768,"data_stored":282656768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284585984,"data_stored":284585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:34.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:34.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'7790", 'reported_seq': 8049, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:31.304292+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:31.304292+0000', 'last_peered': '2024-09-10T20:25:31.304292+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:31.304292+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7790, 'log_dups_size': 0, 'ondisk_log_size': 7790, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35487744, 'num_objects': 4332, 'num_object_clones': 0, 'num_object_copies': 17328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1866, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7790, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 100, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2466}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:34.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:34.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7833", 'reported_seq': 8023, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:33.492310+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:33.492310+0000', 'last_peered': '2024-09-10T20:25:33.492310+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:33.492310+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7833, 'log_dups_size': 0, 'ondisk_log_size': 7833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35281512, 'num_objects': 4308, 'num_object_clones': 0, 'num_object_copies': 17232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1915, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4308, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7832, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2394}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'7787", 'reported_seq': 8076, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:31.460564+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:31.460564+0000', 'last_peered': '2024-09-10T20:25:31.460564+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:31.460564+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7787, 'log_dups_size': 0, 'ondisk_log_size': 7787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35894432, 'num_objects': 4382, 'num_object_clones': 0, 'num_object_copies': 17528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1872, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4382, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7785, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 112, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2512}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7768", 'reported_seq': 7975, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:31.103437+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:31.103437+0000', 'last_peered': '2024-09-10T20:25:31.103437+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:31.103437+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7768, 'log_dups_size': 0, 'ondisk_log_size': 7768, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35471360, 'num_objects': 4330, 'num_object_clones': 0, 'num_object_copies': 17320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1864, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7768, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 74, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2466}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7740", 'reported_seq': 7924, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:30.792147+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:30.792147+0000', 'last_peered': '2024-09-10T20:25:30.792147+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:30.792147+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7740, 'log_dups_size': 0, 'ondisk_log_size': 7740, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35468448, 'num_objects': 4330, 'num_object_clones': 0, 'num_object_copies': 17320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7738, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2428}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'7932", 'reported_seq': 8080, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:30.805084+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:30.805084+0000', 'last_peered': '2024-09-10T20:25:30.805084+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:30.805084+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7932, 'log_dups_size': 0, 'ondisk_log_size': 7932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35722400, 'num_objects': 4361, 'num_object_clones': 0, 'num_object_copies': 17444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2011, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7930, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:34.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7781", 'reported_seq': 8039, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:31.457420+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:31.457420+0000', 'last_peered': '2024-09-10T20:25:31.457420+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:31.457420+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7781, 'log_dups_size': 0, 'ondisk_log_size': 7781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35483376, 'num_objects': 4332, 'num_object_clones': 0, 'num_object_copies': 17328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1866, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7778, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 96, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2469}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'7825", 'reported_seq': 8106, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:31.475375+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:31.475375+0000', 'last_peered': '2024-09-10T20:25:31.475375+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:31.475375+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7825, 'log_dups_size': 0, 'ondisk_log_size': 7825, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35551824, 'num_objects': 4340, 'num_object_clones': 0, 'num_object_copies': 17360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1869, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7824, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7667", 'reported_seq': 7837, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:33.418897+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:33.418897+0000', 'last_peered': '2024-09-10T20:25:33.418897+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:33.418897+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7667, 'log_dups_size': 0, 'ondisk_log_size': 7667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34110032, 'num_objects': 4164, 'num_object_clones': 0, 'num_object_copies': 16656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7666, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'7819", 'reported_seq': 8073, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:31.400914+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:31.400914+0000', 'last_peered': '2024-09-10T20:25:31.400914+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:31.400914+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7819, 'log_dups_size': 0, 'ondisk_log_size': 7819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35017888, 'num_objects': 4275, 'num_object_clones': 0, 'num_object_copies': 17100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7817, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 95, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2393}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'7988", 'reported_seq': 8164, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:33.261709+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:33.261709+0000', 'last_peered': '2024-09-10T20:25:33.261709+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:33.261709+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7988, 'log_dups_size': 0, 'ondisk_log_size': 7988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36541600, 'num_objects': 4461, 'num_object_clones': 0, 'num_object_copies': 17844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1924, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7986, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:34.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7815", 'reported_seq': 8653, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:30.843281+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:30.843281+0000', 'last_peered': '2024-09-10T20:25:30.843281+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:30.843281+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7815, 'log_dups_size': 0, 'ondisk_log_size': 7815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35304608, 'num_objects': 4310, 'num_object_clones': 0, 'num_object_copies': 17240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1603, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7813, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 380, 'num_bytes_recovered': 2859008, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2707}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:34.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:35.258 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:35.258 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22488/277856 objects degraded (8.093%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":424,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69464,"data_bytes":569013256,"bytes_used":2763251712,"bytes_avail":383783804928,"bytes_total":386547056640,"degraded_objects":22488,"degraded_total":277856,"degraded_ratio":0.080934008983070363,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":69594,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":835584,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":232},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:35.271 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:35.271 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:35.825 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:35.825 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:35.854 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-10T20:25:34.004924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569013256,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104881,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125007,"ondisk_log_size":125007,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698488,"kb_used_data":1102072,"kb_used_omap":14,"kb_used_meta":1596337,"kb_avail":374788872,"statfs":{"total":386547056640,"available":383783804928,"internally_reserved":0,"allocated":1128521728,"data_stored":1127669484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1634649654},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,11,7,21,3,0,0,0,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22839296,"num_objects":-2788,"num_object_clones":0,"num_object_copies":-11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006414"},"pg_stats":[{"pgid":"1.f","version":"40'7790","reported_seq":8049,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.304292+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:31.304292+0000","last_peered":"2024-09-10T20:25:31.304292+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:31.304292+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35487744,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7790,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7847","reported_seq":7978,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.295786+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:29.295786+0000","last_peered":"2024-09-10T20:25:29.295786+0000","last_clean":"2024-09-10T20:25:29.295786+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:29.295786+0000","last_undegraded":"2024-09-10T20:25:29.295786+0000","last_fullsized":"2024-09-10T20:25:29.295786+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7847,"log_dups_size":0,"ondisk_log_size":7847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36085760,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7833","reported_seq":8023,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.492310+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:33.492310+0000","last_peered":"2024-09-10T20:25:33.492310+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:33.492310+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7833,"log_dups_size":0,"ondisk_log_size":7833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35281512,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7832,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2394}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7911","reported_seq":12080,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:29.390554+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:29.390554+0000","last_peered":"2024-09-10T20:25:29.390554+0000","last_clean":"2024-09-10T20:25:29.390554+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:29.390554+0000","last_undegraded":"2024-09-10T20:25:29.390554+0000","last_fullsized":"2024-09-10T20:25:29.390554+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36823040,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7787","reported_seq":8076,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.460564+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:31.460564+0000","last_peered":"2024-09-10T20:25:31.460564+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:31.460564+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35894432,"num_objects":4382,"num_object_clones":0,"num_object_copies":17528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7785,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7768","reported_seq":7975,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.103437+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:31.103437+0000","last_peered":"2024-09-10T20:25:31.103437+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:31.103437+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7768,"log_dups_size":0,"ondisk_log_size":7768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35471360,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7768,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7740","reported_seq":7924,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.792147+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:30.792147+0000","last_peered":"2024-09-10T20:25:30.792147+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:30.792147+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7740,"log_dups_size":0,"ondisk_log_size":7740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35468448,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2428}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'7932","reported_seq":8080,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.805084+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:30.805084+0000","last_peered":"2024-09-10T20:25:30.805084+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:30.805084+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722400,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7719","reported_seq":11690,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:31.511812+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:31.511812+0000","last_peered":"2024-09-10T20:25:31.511812+0000","last_clean":"2024-09-10T20:25:31.511812+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:31.511812+0000","last_undegraded":"2024-09-10T20:25:31.511812+0000","last_fullsized":"2024-09-10T20:25:31.511812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35076688,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7785","reported_seq":11755,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:33.456185+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:33.456185+0000","last_peered":"2024-09-10T20:25:33.456185+0000","last_clean":"2024-09-10T20:25:33.456185+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:33.456185+0000","last_undegraded":"2024-09-10T20:25:33.456185+0000","last_fullsized":"2024-09-10T20:25:33.456185+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35692544,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7781","reported_seq":8039,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.457420+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:31.457420+0000","last_peered":"2024-09-10T20:25:31.457420+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:31.457420+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483376,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7825","reported_seq":8106,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.475375+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:31.475375+0000","last_peered":"2024-09-10T20:25:31.475375+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:31.475375+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7825,"log_dups_size":0,"ondisk_log_size":7825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35551824,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7667","reported_seq":7837,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.418897+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:33.418897+0000","last_peered":"2024-09-10T20:25:33.418897+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:33.418897+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34110032,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2255}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7819","reported_seq":8073,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:31.400914+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:31.400914+0000","last_peered":"2024-09-10T20:25:31.400914+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:31.400914+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35017888,"num_objects":4275,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2393}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7988","reported_seq":8164,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.261709+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:33.261709+0000","last_peered":"2024-09-10T20:25:33.261709+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:33.261709+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541600,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7815","reported_seq":8653,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:30.843281+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:30.843281+0000","last_peered":"2024-09-10T20:25:30.843281+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:30.843281+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7815,"log_dups_size":0,"ondisk_log_size":7815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304608,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7813,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2707}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569013256,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104881,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":2859008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126354944,"data_stored":1126354944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125007,"ondisk_log_size":125007,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707092,"kb_used_data":278340,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93664748,"statfs":{"total":96636764160,"available":95912701952,"internally_reserved":0,"allocated":285020160,"data_stored":284807099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":276976,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":283623424,"data_stored":283410363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,5,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574316,"kb_used_data":267740,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93797524,"statfs":{"total":96636764160,"available":96048664576,"internally_reserved":0,"allocated":274165760,"data_stored":273952699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707704,"kb_used_data":279016,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93664136,"statfs":{"total":96636764160,"available":95912075264,"internally_reserved":0,"allocated":285712384,"data_stored":285499323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285282304,"data_stored":285282304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273829888,"data_stored":273829888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282656768,"data_stored":282656768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284585984,"data_stored":284585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:38.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:39.433 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:39.458 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-10T20:25:38.006279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561878024,"num_objects":68593,"num_object_clones":0,"num_object_copies":274372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68593,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105752,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1321,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125878,"ondisk_log_size":125878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2719204,"kb_used_data":1092256,"kb_used_omap":14,"kb_used_meta":1626865,"kb_avail":374768156,"statfs":{"total":386547056640,"available":383762591744,"internally_reserved":0,"allocated":1118470144,"data_stored":1117617900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1665910326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,6,14,12,5,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24363008,"num_objects":-2974,"num_object_clones":0,"num_object_copies":-11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2974,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004372"},"pg_stats":[{"pgid":"1.f","version":"40'7867","reported_seq":8135,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.557963+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:36.557963+0000","last_peered":"2024-09-10T20:25:36.557963+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:36.557963+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7867,"log_dups_size":0,"ondisk_log_size":7867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34856960,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7867,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'7930","reported_seq":8060,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:34.366564+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:34.366564+0000","last_peered":"2024-09-10T20:25:34.366564+0000","last_clean":"2024-09-10T20:25:34.366564+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:34.366564+0000","last_undegraded":"2024-09-10T20:25:34.366564+0000","last_fullsized":"2024-09-10T20:25:34.366564+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35405824,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4371,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7833","reported_seq":8023,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.492310+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:33.492310+0000","last_peered":"2024-09-10T20:25:33.492310+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:33.492310+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7833,"log_dups_size":0,"ondisk_log_size":7833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35281512,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7832,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2394}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'7990","reported_seq":12159,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:34.316646+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:34.316646+0000","last_peered":"2024-09-10T20:25:34.316646+0000","last_clean":"2024-09-10T20:25:34.316646+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:34.316646+0000","last_undegraded":"2024-09-10T20:25:34.316646+0000","last_fullsized":"2024-09-10T20:25:34.316646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36175872,"num_objects":4416,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7852","reported_seq":8143,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.426925+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:36.426925+0000","last_peered":"2024-09-10T20:25:36.426925+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:36.426925+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361952,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2448}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7840","reported_seq":8047,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.497641+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:36.497641+0000","last_peered":"2024-09-10T20:25:36.497641+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:36.497641+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7840,"log_dups_size":0,"ondisk_log_size":7840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":4258,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7840,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7809","reported_seq":7998,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.882655+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:35.882655+0000","last_peered":"2024-09-10T20:25:35.882655+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:35.882655+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7809,"log_dups_size":0,"ondisk_log_size":7809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34903200,"num_objects":4261,"num_object_clones":0,"num_object_copies":17044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7807,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8012","reported_seq":8173,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.829089+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:35.829089+0000","last_peered":"2024-09-10T20:25:35.829089+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:35.829089+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8012,"log_dups_size":0,"ondisk_log_size":8012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35067040,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7785","reported_seq":11756,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:36.382102+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:36.382102+0000","last_peered":"2024-09-10T20:25:36.382102+0000","last_clean":"2024-09-10T20:25:36.382102+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:36.382102+0000","last_undegraded":"2024-09-10T20:25:36.382102+0000","last_fullsized":"2024-09-10T20:25:36.382102+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34536016,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7785","reported_seq":11755,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:33.456185+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:33.456185+0000","last_peered":"2024-09-10T20:25:33.456185+0000","last_clean":"2024-09-10T20:25:33.456185+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:33.456185+0000","last_undegraded":"2024-09-10T20:25:33.456185+0000","last_fullsized":"2024-09-10T20:25:33.456185+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35692544,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7850","reported_seq":8110,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.465969+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:36.465969+0000","last_peered":"2024-09-10T20:25:36.465969+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:36.465969+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34918128,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7847,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7896","reported_seq":8178,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.536676+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:36.536676+0000","last_peered":"2024-09-10T20:25:36.536676+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:36.536676+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34970192,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7895,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7667","reported_seq":7837,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.418897+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:33.418897+0000","last_peered":"2024-09-10T20:25:33.418897+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:33.418897+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34110032,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2255}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7889","reported_seq":8147,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.471019+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:36.471019+0000","last_peered":"2024-09-10T20:25:36.471019+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:36.471019+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444448,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2325}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'7988","reported_seq":8164,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:33.261709+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:33.261709+0000","last_peered":"2024-09-10T20:25:33.261709+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:33.261709+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541600,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7885","reported_seq":8952,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.852735+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:35.852735+0000","last_peered":"2024-09-10T20:25:35.852735+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:35.852735+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34731168,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":561878024,"num_objects":68593,"num_object_clones":0,"num_object_copies":274372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68593,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105752,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1321,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1115906048,"data_stored":1115906048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125878,"ondisk_log_size":125878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706748,"kb_used_data":273836,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93665092,"statfs":{"total":96636764160,"available":95913054208,"internally_reserved":0,"allocated":280408064,"data_stored":280195003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":276976,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":283623424,"data_stored":283410363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,5,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595644,"kb_used_data":266860,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93776196,"statfs":{"total":96636764160,"available":96026824704,"internally_reserved":0,"allocated":273264640,"data_stored":273051579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707436,"kb_used_data":274584,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93664404,"statfs":{"total":96636764160,"available":95912349696,"internally_reserved":0,"allocated":281174016,"data_stored":280960955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280641536,"data_stored":280641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272691200,"data_stored":272691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282656768,"data_stored":282656768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279916544,"data_stored":279916544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:39.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:40.083 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:40.083 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:40.106 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-10T20:25:40.007134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558289928,"num_objects":68155,"num_object_clones":0,"num_object_copies":272620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106190,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126316,"ondisk_log_size":126316,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705092,"kb_used_data":1086460,"kb_used_omap":14,"kb_used_meta":1618545,"kb_avail":374782268,"statfs":{"total":386547056640,"available":383777042432,"internally_reserved":0,"allocated":1112535040,"data_stored":1111682796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1657390646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,7,11,10,5,0,0,0,0,1,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23658496,"num_objects":-2888,"num_object_clones":0,"num_object_copies":-11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2888,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004544"},"pg_stats":[{"pgid":"1.f","version":"40'7867","reported_seq":8135,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.557963+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:36.557963+0000","last_peered":"2024-09-10T20:25:36.557963+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:36.557963+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7867,"log_dups_size":0,"ondisk_log_size":7867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34856960,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7867,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8002","reported_seq":8133,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.434898+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:39.434898+0000","last_peered":"2024-09-10T20:25:39.434898+0000","last_clean":"2024-09-10T20:25:39.434898+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:39.434898+0000","last_undegraded":"2024-09-10T20:25:39.434898+0000","last_fullsized":"2024-09-10T20:25:39.434898+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34816000,"num_objects":4250,"num_object_clones":0,"num_object_copies":17000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7910","reported_seq":8104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:38.561843+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:38.561843+0000","last_peered":"2024-09-10T20:25:38.561843+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:38.561843+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34650728,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7909,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8077","reported_seq":12243,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.414983+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:39.414983+0000","last_peered":"2024-09-10T20:25:39.414983+0000","last_clean":"2024-09-10T20:25:39.414983+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:39.414983+0000","last_undegraded":"2024-09-10T20:25:39.414983+0000","last_fullsized":"2024-09-10T20:25:39.414983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35463168,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7852","reported_seq":8143,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.426925+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:36.426925+0000","last_peered":"2024-09-10T20:25:36.426925+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:36.426925+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361952,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2448}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7840","reported_seq":8047,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.497641+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:36.497641+0000","last_peered":"2024-09-10T20:25:36.497641+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:36.497641+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7840,"log_dups_size":0,"ondisk_log_size":7840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":4258,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7840,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7809","reported_seq":7998,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.882655+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:35.882655+0000","last_peered":"2024-09-10T20:25:35.882655+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:35.882655+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7809,"log_dups_size":0,"ondisk_log_size":7809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34903200,"num_objects":4261,"num_object_clones":0,"num_object_copies":17044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7807,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8012","reported_seq":8173,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.829089+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:35.829089+0000","last_peered":"2024-09-10T20:25:35.829089+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:35.829089+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8012,"log_dups_size":0,"ondisk_log_size":8012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35067040,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7785","reported_seq":11756,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:36.382102+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:36.382102+0000","last_peered":"2024-09-10T20:25:36.382102+0000","last_clean":"2024-09-10T20:25:36.382102+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:36.382102+0000","last_undegraded":"2024-09-10T20:25:36.382102+0000","last_fullsized":"2024-09-10T20:25:36.382102+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34536016,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7849","reported_seq":11819,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:38.530801+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:38.530801+0000","last_peered":"2024-09-10T20:25:38.530801+0000","last_clean":"2024-09-10T20:25:38.530801+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:38.530801+0000","last_undegraded":"2024-09-10T20:25:38.530801+0000","last_fullsized":"2024-09-10T20:25:38.530801+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35168256,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":487,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7850","reported_seq":8110,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.465969+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:36.465969+0000","last_peered":"2024-09-10T20:25:36.465969+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:36.465969+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34918128,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7847,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7896","reported_seq":8178,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.536676+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:36.536676+0000","last_peered":"2024-09-10T20:25:36.536676+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:36.536676+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34970192,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7895,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7738","reported_seq":7912,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:38.523707+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:38.523707+0000","last_peered":"2024-09-10T20:25:38.523707+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:38.523707+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528400,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7737,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7889","reported_seq":8147,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.471019+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:36.471019+0000","last_peered":"2024-09-10T20:25:36.471019+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:36.471019+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444448,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2325}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8055","reported_seq":8241,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:38.555007+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:38.555007+0000","last_peered":"2024-09-10T20:25:38.555007+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:38.555007+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8055,"log_dups_size":0,"ondisk_log_size":8055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35992736,"num_objects":4394,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8053,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7885","reported_seq":8952,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.852735+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:35.852735+0000","last_peered":"2024-09-10T20:25:35.852735+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:35.852735+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34731168,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558289928,"num_objects":68155,"num_object_clones":0,"num_object_copies":272620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106190,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110228992,"data_stored":1110228992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126316,"ondisk_log_size":126316,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706748,"kb_used_data":273836,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93665092,"statfs":{"total":96636764160,"available":95913054208,"internally_reserved":0,"allocated":280408064,"data_stored":280195003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708744,"kb_used_data":272184,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93663096,"statfs":{"total":96636764160,"available":95911010304,"internally_reserved":0,"allocated":278716416,"data_stored":278503355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,4,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582164,"kb_used_data":265856,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93789676,"statfs":{"total":96636764160,"available":96040628224,"internally_reserved":0,"allocated":272236544,"data_stored":272023483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707436,"kb_used_data":274584,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93664404,"statfs":{"total":96636764160,"available":95912349696,"internally_reserved":0,"allocated":281174016,"data_stored":280960955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280641536,"data_stored":280641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271777792,"data_stored":271777792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277893120,"data_stored":277893120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279916544,"data_stored":279916544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:40.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'7867", 'reported_seq': 8135, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:36.557963+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:36.557963+0000', 'last_peered': '2024-09-10T20:25:36.557963+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:36.557963+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7867, 'log_dups_size': 0, 'ondisk_log_size': 7867, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34856960, 'num_objects': 4255, 'num_object_clones': 0, 'num_object_copies': 17020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7867, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 105, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7910", 'reported_seq': 8104, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:38.561843+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:38.561843+0000', 'last_peered': '2024-09-10T20:25:38.561843+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:38.561843+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7910, 'log_dups_size': 0, 'ondisk_log_size': 7910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34650728, 'num_objects': 4231, 'num_object_clones': 0, 'num_object_copies': 16924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1912, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4231, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7909, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2320}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'7852", 'reported_seq': 8143, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:36.426925+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:36.426925+0000', 'last_peered': '2024-09-10T20:25:36.426925+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:36.426925+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7852, 'log_dups_size': 0, 'ondisk_log_size': 7852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35361952, 'num_objects': 4317, 'num_object_clones': 0, 'num_object_copies': 17268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1871, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7850, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 113, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2448}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7840", 'reported_seq': 8047, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:36.497641+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:36.497641+0000', 'last_peered': '2024-09-10T20:25:36.497641+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:36.497641+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7840, 'log_dups_size': 0, 'ondisk_log_size': 7840, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34881536, 'num_objects': 4258, 'num_object_clones': 0, 'num_object_copies': 17032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1864, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7840, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 74, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7809", 'reported_seq': 7998, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:35.882655+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:35.882655+0000', 'last_peered': '2024-09-10T20:25:35.882655+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:35.882655+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7809, 'log_dups_size': 0, 'ondisk_log_size': 7809, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34903200, 'num_objects': 4261, 'num_object_clones': 0, 'num_object_copies': 17044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1902, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7807, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:40.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8012", 'reported_seq': 8173, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:35.829089+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:35.829089+0000', 'last_peered': '2024-09-10T20:25:35.829089+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:35.829089+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8012, 'log_dups_size': 0, 'ondisk_log_size': 8012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35067040, 'num_objects': 4281, 'num_object_clones': 0, 'num_object_copies': 17124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8010, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2278}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7850", 'reported_seq': 8110, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:36.465969+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:36.465969+0000', 'last_peered': '2024-09-10T20:25:36.465969+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:36.465969+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7850, 'log_dups_size': 0, 'ondisk_log_size': 7850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34918128, 'num_objects': 4263, 'num_object_clones': 0, 'num_object_copies': 17052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1865, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7847, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 97, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2401}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'7896", 'reported_seq': 8178, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:36.536676+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:36.536676+0000', 'last_peered': '2024-09-10T20:25:36.536676+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:36.536676+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7896, 'log_dups_size': 0, 'ondisk_log_size': 7896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34970192, 'num_objects': 4269, 'num_object_clones': 0, 'num_object_copies': 17076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1868, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7895, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 112, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7738", 'reported_seq': 7912, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:38.523707+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:38.523707+0000', 'last_peered': '2024-09-10T20:25:38.523707+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:38.523707+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7738, 'log_dups_size': 0, 'ondisk_log_size': 7738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33528400, 'num_objects': 4093, 'num_object_clones': 0, 'num_object_copies': 16372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1908, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7737, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'7889", 'reported_seq': 8147, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:36.471019+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:36.471019+0000', 'last_peered': '2024-09-10T20:25:36.471019+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:36.471019+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7889, 'log_dups_size': 0, 'ondisk_log_size': 7889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34444448, 'num_objects': 4205, 'num_object_clones': 0, 'num_object_copies': 16820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1882, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7887, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 97, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2325}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8055", 'reported_seq': 8241, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:38.555007+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:38.555007+0000', 'last_peered': '2024-09-10T20:25:38.555007+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:38.555007+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8055, 'log_dups_size': 0, 'ondisk_log_size': 8055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35992736, 'num_objects': 4394, 'num_object_clones': 0, 'num_object_copies': 17576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1919, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8053, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 47, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:40.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:40.110 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7885", 'reported_seq': 8952, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:35.852735+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:35.852735+0000', 'last_peered': '2024-09-10T20:25:35.852735+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:35.852735+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7885, 'log_dups_size': 0, 'ondisk_log_size': 7885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34731168, 'num_objects': 4240, 'num_object_clones': 0, 'num_object_copies': 16960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1488, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7883, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 495, 'num_bytes_recovered': 3768320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:40.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:40.881 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22345/272620 objects degraded (8.196%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":429,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68155,"data_bytes":558289928,"bytes_used":2770014208,"bytes_avail":383777042432,"bytes_total":386547056640,"degraded_objects":22345,"degraded_total":272620,"degraded_ratio":0.081963905802949155,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":111232,"recovering_keys_per_sec":0,"num_objects_recovered":230,"num_bytes_recovered":1335296,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:40.909 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:40.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:41.529 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:41.530 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:41.558 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-10T20:25:40.007134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558289928,"num_objects":68155,"num_object_clones":0,"num_object_copies":272620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106190,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126316,"ondisk_log_size":126316,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705092,"kb_used_data":1086460,"kb_used_omap":14,"kb_used_meta":1618545,"kb_avail":374782268,"statfs":{"total":386547056640,"available":383777042432,"internally_reserved":0,"allocated":1112535040,"data_stored":1111682796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1657390646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,7,11,10,5,0,0,0,0,1,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23658496,"num_objects":-2888,"num_object_clones":0,"num_object_copies":-11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2888,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004544"},"pg_stats":[{"pgid":"1.f","version":"40'7867","reported_seq":8135,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.557963+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:36.557963+0000","last_peered":"2024-09-10T20:25:36.557963+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:36.557963+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7867,"log_dups_size":0,"ondisk_log_size":7867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34856960,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7867,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8002","reported_seq":8133,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.434898+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:39.434898+0000","last_peered":"2024-09-10T20:25:39.434898+0000","last_clean":"2024-09-10T20:25:39.434898+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:39.434898+0000","last_undegraded":"2024-09-10T20:25:39.434898+0000","last_fullsized":"2024-09-10T20:25:39.434898+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34816000,"num_objects":4250,"num_object_clones":0,"num_object_copies":17000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7910","reported_seq":8104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:38.561843+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:38.561843+0000","last_peered":"2024-09-10T20:25:38.561843+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:38.561843+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34650728,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7909,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8077","reported_seq":12243,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.414983+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:39.414983+0000","last_peered":"2024-09-10T20:25:39.414983+0000","last_clean":"2024-09-10T20:25:39.414983+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:39.414983+0000","last_undegraded":"2024-09-10T20:25:39.414983+0000","last_fullsized":"2024-09-10T20:25:39.414983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35463168,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7852","reported_seq":8143,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.426925+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:36.426925+0000","last_peered":"2024-09-10T20:25:36.426925+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:36.426925+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361952,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2448}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7840","reported_seq":8047,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.497641+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:36.497641+0000","last_peered":"2024-09-10T20:25:36.497641+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:36.497641+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7840,"log_dups_size":0,"ondisk_log_size":7840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":4258,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7840,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7809","reported_seq":7998,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.882655+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:35.882655+0000","last_peered":"2024-09-10T20:25:35.882655+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:35.882655+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7809,"log_dups_size":0,"ondisk_log_size":7809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34903200,"num_objects":4261,"num_object_clones":0,"num_object_copies":17044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7807,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8012","reported_seq":8173,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.829089+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:35.829089+0000","last_peered":"2024-09-10T20:25:35.829089+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:35.829089+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8012,"log_dups_size":0,"ondisk_log_size":8012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35067040,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7785","reported_seq":11756,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:36.382102+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:36.382102+0000","last_peered":"2024-09-10T20:25:36.382102+0000","last_clean":"2024-09-10T20:25:36.382102+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:36.382102+0000","last_undegraded":"2024-09-10T20:25:36.382102+0000","last_fullsized":"2024-09-10T20:25:36.382102+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34536016,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7849","reported_seq":11819,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:38.530801+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:38.530801+0000","last_peered":"2024-09-10T20:25:38.530801+0000","last_clean":"2024-09-10T20:25:38.530801+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:38.530801+0000","last_undegraded":"2024-09-10T20:25:38.530801+0000","last_fullsized":"2024-09-10T20:25:38.530801+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35168256,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":487,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7850","reported_seq":8110,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.465969+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:36.465969+0000","last_peered":"2024-09-10T20:25:36.465969+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:36.465969+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34918128,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7847,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7896","reported_seq":8178,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.536676+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:36.536676+0000","last_peered":"2024-09-10T20:25:36.536676+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:36.536676+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34970192,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7895,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7738","reported_seq":7912,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:38.523707+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:38.523707+0000","last_peered":"2024-09-10T20:25:38.523707+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:38.523707+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528400,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7737,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7889","reported_seq":8147,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:36.471019+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:36.471019+0000","last_peered":"2024-09-10T20:25:36.471019+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:36.471019+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444448,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2325}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8055","reported_seq":8241,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:38.555007+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:38.555007+0000","last_peered":"2024-09-10T20:25:38.555007+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:38.555007+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8055,"log_dups_size":0,"ondisk_log_size":8055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35992736,"num_objects":4394,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8053,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7885","reported_seq":8952,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:35.852735+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:35.852735+0000","last_peered":"2024-09-10T20:25:35.852735+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:35.852735+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34731168,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558289928,"num_objects":68155,"num_object_clones":0,"num_object_copies":272620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106190,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110228992,"data_stored":1110228992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126316,"ondisk_log_size":126316,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706748,"kb_used_data":273836,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93665092,"statfs":{"total":96636764160,"available":95913054208,"internally_reserved":0,"allocated":280408064,"data_stored":280195003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708744,"kb_used_data":272184,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93663096,"statfs":{"total":96636764160,"available":95911010304,"internally_reserved":0,"allocated":278716416,"data_stored":278503355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,4,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582164,"kb_used_data":265856,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93789676,"statfs":{"total":96636764160,"available":96040628224,"internally_reserved":0,"allocated":272236544,"data_stored":272023483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707436,"kb_used_data":274584,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93664404,"statfs":{"total":96636764160,"available":95912349696,"internally_reserved":0,"allocated":281174016,"data_stored":280960955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280641536,"data_stored":280641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271777792,"data_stored":271777792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277893120,"data_stored":277893120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279916544,"data_stored":279916544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:44.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:45.150 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:45.151 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:45.176 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-10T20:25:44.008621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":549712904,"num_objects":67108,"num_object_clones":0,"num_object_copies":268432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107237,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1448,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127363,"ondisk_log_size":127363,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2686628,"kb_used_data":1072136,"kb_used_omap":14,"kb_used_meta":1614385,"kb_avail":374800732,"statfs":{"total":386547056640,"available":383795949568,"internally_reserved":0,"allocated":1097867264,"data_stored":1097015020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1653130806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,5,10,14,6,0,3,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21921792,"num_objects":-2676,"num_object_clones":0,"num_object_copies":-10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2676,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004543"},"pg_stats":[{"pgid":"1.f","version":"40'7936","reported_seq":8211,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.649759+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:41.649759+0000","last_peered":"2024-09-10T20:25:41.649759+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:41.649759+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34291712,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7936,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8002","reported_seq":8133,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.434898+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:39.434898+0000","last_peered":"2024-09-10T20:25:39.434898+0000","last_clean":"2024-09-10T20:25:39.434898+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:39.434898+0000","last_undegraded":"2024-09-10T20:25:39.434898+0000","last_fullsized":"2024-09-10T20:25:39.434898+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34816000,"num_objects":4250,"num_object_clones":0,"num_object_copies":17000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7985","reported_seq":8181,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.586734+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:43.586734+0000","last_peered":"2024-09-10T20:25:43.586734+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:43.586734+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34036328,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7984,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2245}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8077","reported_seq":12243,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.414983+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:39.414983+0000","last_peered":"2024-09-10T20:25:39.414983+0000","last_clean":"2024-09-10T20:25:39.414983+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:39.414983+0000","last_undegraded":"2024-09-10T20:25:39.414983+0000","last_fullsized":"2024-09-10T20:25:39.414983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35463168,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7960","reported_seq":8256,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.651448+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:41.651448+0000","last_peered":"2024-09-10T20:25:41.651448+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:41.651448+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34477216,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7906","reported_seq":8119,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.556138+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:41.556138+0000","last_peered":"2024-09-10T20:25:41.556138+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:41.556138+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7879","reported_seq":8068,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:40.783091+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:40.783091+0000","last_peered":"2024-09-10T20:25:40.783091+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:40.783091+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7879,"log_dups_size":0,"ondisk_log_size":7879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7877,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2291}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8082","reported_seq":8245,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:40.849651+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:40.849651+0000","last_peered":"2024-09-10T20:25:40.849651+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:40.849651+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34493600,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7843","reported_seq":11814,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:41.463728+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:41.463728+0000","last_peered":"2024-09-10T20:25:41.463728+0000","last_clean":"2024-09-10T20:25:41.463728+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:41.463728+0000","last_undegraded":"2024-09-10T20:25:41.463728+0000","last_fullsized":"2024-09-10T20:25:41.463728+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7919","reported_seq":11889,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:43.469626+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:43.469626+0000","last_peered":"2024-09-10T20:25:43.469626+0000","last_clean":"2024-09-10T20:25:43.469626+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:43.469626+0000","last_undegraded":"2024-09-10T20:25:43.469626+0000","last_fullsized":"2024-09-10T20:25:43.469626+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7919,"log_dups_size":0,"ondisk_log_size":7919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34594816,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":557,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7927","reported_seq":8191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.492790+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:41.492790+0000","last_peered":"2024-09-10T20:25:41.492790+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:41.492790+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7927,"log_dups_size":0,"ondisk_log_size":7927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34287344,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7980","reported_seq":8271,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.684140+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:41.684140+0000","last_peered":"2024-09-10T20:25:41.684140+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:41.684140+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7980,"log_dups_size":0,"ondisk_log_size":7980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34282064,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2322}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7810","reported_seq":7987,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.428083+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:43.428083+0000","last_peered":"2024-09-10T20:25:43.428083+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:43.428083+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7810,"log_dups_size":0,"ondisk_log_size":7810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32938576,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7809,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7952","reported_seq":8212,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.608991+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:41.608991+0000","last_peered":"2024-09-10T20:25:41.608991+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:41.608991+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7952,"log_dups_size":0,"ondisk_log_size":7952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33928352,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7950,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8143","reported_seq":8342,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.428058+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:43.428058+0000","last_peered":"2024-09-10T20:25:43.428058+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:43.428058+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8143,"log_dups_size":0,"ondisk_log_size":8143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271840,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2395}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7962","reported_seq":9212,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:40.881571+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:40.881571+0000","last_peered":"2024-09-10T20:25:40.881571+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:40.881571+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7962,"log_dups_size":0,"ondisk_log_size":7962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34100384,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7960,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":587,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":549712904,"num_objects":67108,"num_object_clones":0,"num_object_copies":268432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107237,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1448,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095368704,"data_stored":1095368704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127363,"ondisk_log_size":127363,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706228,"kb_used_data":269144,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93665612,"statfs":{"total":96636764160,"available":95913586688,"internally_reserved":0,"allocated":275603456,"data_stored":275390395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,1,1,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691384,"kb_used_data":267292,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93680456,"statfs":{"total":96636764160,"available":95928786944,"internally_reserved":0,"allocated":273707008,"data_stored":273493947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,4,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582164,"kb_used_data":265856,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93789676,"statfs":{"total":96636764160,"available":96040628224,"internally_reserved":0,"allocated":272236544,"data_stored":272023483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706852,"kb_used_data":269844,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93664988,"statfs":{"total":96636764160,"available":95912947712,"internally_reserved":0,"allocated":276320256,"data_stored":276107195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275738624,"data_stored":275738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271777792,"data_stored":271777792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272904192,"data_stored":272904192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274948096,"data_stored":274948096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:45.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:45.727 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:45.728 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:45.747 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-10T20:25:44.008621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":549712904,"num_objects":67108,"num_object_clones":0,"num_object_copies":268432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107237,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1448,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127363,"ondisk_log_size":127363,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2686628,"kb_used_data":1072136,"kb_used_omap":14,"kb_used_meta":1614385,"kb_avail":374800732,"statfs":{"total":386547056640,"available":383795949568,"internally_reserved":0,"allocated":1097867264,"data_stored":1097015020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1653130806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,5,10,14,6,0,3,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21921792,"num_objects":-2676,"num_object_clones":0,"num_object_copies":-10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2676,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004543"},"pg_stats":[{"pgid":"1.f","version":"40'7936","reported_seq":8211,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.649759+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:41.649759+0000","last_peered":"2024-09-10T20:25:41.649759+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:41.649759+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34291712,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7936,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8002","reported_seq":8133,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.434898+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:39.434898+0000","last_peered":"2024-09-10T20:25:39.434898+0000","last_clean":"2024-09-10T20:25:39.434898+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:39.434898+0000","last_undegraded":"2024-09-10T20:25:39.434898+0000","last_fullsized":"2024-09-10T20:25:39.434898+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34816000,"num_objects":4250,"num_object_clones":0,"num_object_copies":17000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7985","reported_seq":8181,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.586734+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:43.586734+0000","last_peered":"2024-09-10T20:25:43.586734+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:43.586734+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34036328,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7984,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2245}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8077","reported_seq":12243,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:39.414983+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:39.414983+0000","last_peered":"2024-09-10T20:25:39.414983+0000","last_clean":"2024-09-10T20:25:39.414983+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:39.414983+0000","last_undegraded":"2024-09-10T20:25:39.414983+0000","last_fullsized":"2024-09-10T20:25:39.414983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35463168,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7960","reported_seq":8256,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.651448+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:41.651448+0000","last_peered":"2024-09-10T20:25:41.651448+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:41.651448+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34477216,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7906","reported_seq":8119,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.556138+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:41.556138+0000","last_peered":"2024-09-10T20:25:41.556138+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:41.556138+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7879","reported_seq":8068,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:40.783091+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:40.783091+0000","last_peered":"2024-09-10T20:25:40.783091+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:40.783091+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7879,"log_dups_size":0,"ondisk_log_size":7879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7877,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2291}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8082","reported_seq":8245,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:40.849651+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:40.849651+0000","last_peered":"2024-09-10T20:25:40.849651+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:40.849651+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34493600,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7843","reported_seq":11814,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:41.463728+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:41.463728+0000","last_peered":"2024-09-10T20:25:41.463728+0000","last_clean":"2024-09-10T20:25:41.463728+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:41.463728+0000","last_undegraded":"2024-09-10T20:25:41.463728+0000","last_fullsized":"2024-09-10T20:25:41.463728+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7919","reported_seq":11889,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:43.469626+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:43.469626+0000","last_peered":"2024-09-10T20:25:43.469626+0000","last_clean":"2024-09-10T20:25:43.469626+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:43.469626+0000","last_undegraded":"2024-09-10T20:25:43.469626+0000","last_fullsized":"2024-09-10T20:25:43.469626+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7919,"log_dups_size":0,"ondisk_log_size":7919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34594816,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":557,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7927","reported_seq":8191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.492790+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:41.492790+0000","last_peered":"2024-09-10T20:25:41.492790+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:41.492790+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7927,"log_dups_size":0,"ondisk_log_size":7927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34287344,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7980","reported_seq":8271,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.684140+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:41.684140+0000","last_peered":"2024-09-10T20:25:41.684140+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:41.684140+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7980,"log_dups_size":0,"ondisk_log_size":7980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34282064,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2322}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7810","reported_seq":7987,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.428083+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:43.428083+0000","last_peered":"2024-09-10T20:25:43.428083+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:43.428083+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7810,"log_dups_size":0,"ondisk_log_size":7810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32938576,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7809,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7952","reported_seq":8212,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.608991+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:41.608991+0000","last_peered":"2024-09-10T20:25:41.608991+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:41.608991+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7952,"log_dups_size":0,"ondisk_log_size":7952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33928352,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7950,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8143","reported_seq":8342,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.428058+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:43.428058+0000","last_peered":"2024-09-10T20:25:43.428058+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:43.428058+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8143,"log_dups_size":0,"ondisk_log_size":8143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271840,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2395}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'7962","reported_seq":9212,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:40.881571+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:40.881571+0000","last_peered":"2024-09-10T20:25:40.881571+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:40.881571+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7962,"log_dups_size":0,"ondisk_log_size":7962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34100384,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7960,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":587,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":549712904,"num_objects":67108,"num_object_clones":0,"num_object_copies":268432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107237,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1448,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095368704,"data_stored":1095368704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127363,"ondisk_log_size":127363,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706228,"kb_used_data":269144,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93665612,"statfs":{"total":96636764160,"available":95913586688,"internally_reserved":0,"allocated":275603456,"data_stored":275390395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,1,1,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691384,"kb_used_data":267292,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93680456,"statfs":{"total":96636764160,"available":95928786944,"internally_reserved":0,"allocated":273707008,"data_stored":273493947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,4,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582164,"kb_used_data":265856,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93789676,"statfs":{"total":96636764160,"available":96040628224,"internally_reserved":0,"allocated":272236544,"data_stored":272023483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706852,"kb_used_data":269844,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93664988,"statfs":{"total":96636764160,"available":95912947712,"internally_reserved":0,"allocated":276320256,"data_stored":276107195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275738624,"data_stored":275738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271777792,"data_stored":271777792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272904192,"data_stored":272904192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274948096,"data_stored":274948096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'7936", 'reported_seq': 8211, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:41.649759+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:41.649759+0000', 'last_peered': '2024-09-10T20:25:41.649759+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:41.649759+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7936, 'log_dups_size': 0, 'ondisk_log_size': 7936, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34291712, 'num_objects': 4186, 'num_object_clones': 0, 'num_object_copies': 16744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1858, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7936, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 108, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'7985", 'reported_seq': 8181, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:43.586734+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:43.586734+0000', 'last_peered': '2024-09-10T20:25:43.586734+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:43.586734+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7985, 'log_dups_size': 0, 'ondisk_log_size': 7985, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34036328, 'num_objects': 4156, 'num_object_clones': 0, 'num_object_copies': 16624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1912, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4156, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7984, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2245}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'7960", 'reported_seq': 8256, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:41.651448+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:41.651448+0000', 'last_peered': '2024-09-10T20:25:41.651448+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:41.651448+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7960, 'log_dups_size': 0, 'ondisk_log_size': 7960, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34477216, 'num_objects': 4209, 'num_object_clones': 0, 'num_object_copies': 16836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1867, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7958, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 117, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2344}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:45.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7906", 'reported_seq': 8119, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:41.556138+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:41.556138+0000', 'last_peered': '2024-09-10T20:25:41.556138+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:41.556138+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7906, 'log_dups_size': 0, 'ondisk_log_size': 7906, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34340864, 'num_objects': 4192, 'num_object_clones': 0, 'num_object_copies': 16768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7906, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2331}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7879", 'reported_seq': 8068, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:40.783091+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:40.783091+0000', 'last_peered': '2024-09-10T20:25:40.783091+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:40.783091+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7879, 'log_dups_size': 0, 'ondisk_log_size': 7879, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34329760, 'num_objects': 4191, 'num_object_clones': 0, 'num_object_copies': 16764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1902, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7877, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2291}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8082", 'reported_seq': 8245, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:40.849651+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:40.849651+0000', 'last_peered': '2024-09-10T20:25:40.849651+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:40.849651+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8082, 'log_dups_size': 0, 'ondisk_log_size': 8082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34493600, 'num_objects': 4211, 'num_object_clones': 0, 'num_object_copies': 16844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8080, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7927", 'reported_seq': 8191, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:41.492790+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:41.492790+0000', 'last_peered': '2024-09-10T20:25:41.492790+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:41.492790+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7927, 'log_dups_size': 0, 'ondisk_log_size': 7927, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34287344, 'num_objects': 4186, 'num_object_clones': 0, 'num_object_copies': 16744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1863, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7924, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2326}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'7980", 'reported_seq': 8271, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:41.684140+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:41.684140+0000', 'last_peered': '2024-09-10T20:25:41.684140+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:41.684140+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7980, 'log_dups_size': 0, 'ondisk_log_size': 7980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34282064, 'num_objects': 4185, 'num_object_clones': 0, 'num_object_copies': 16740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1864, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7979, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2322}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:45.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7810", 'reported_seq': 7987, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:43.428083+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:43.428083+0000', 'last_peered': '2024-09-10T20:25:43.428083+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:43.428083+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7810, 'log_dups_size': 0, 'ondisk_log_size': 7810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32938576, 'num_objects': 4021, 'num_object_clones': 0, 'num_object_copies': 16084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1907, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7809, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2115}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:45.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:45.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'7952", 'reported_seq': 8212, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:41.608991+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:41.608991+0000', 'last_peered': '2024-09-10T20:25:41.608991+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:41.608991+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7952, 'log_dups_size': 0, 'ondisk_log_size': 7952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33928352, 'num_objects': 4142, 'num_object_clones': 0, 'num_object_copies': 16568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7950, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:45.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:45.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8143", 'reported_seq': 8342, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:43.428058+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:43.428058+0000', 'last_peered': '2024-09-10T20:25:43.428058+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:43.428058+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8143, 'log_dups_size': 0, 'ondisk_log_size': 8143, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35271840, 'num_objects': 4306, 'num_object_clones': 0, 'num_object_copies': 17224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1913, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8141, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:45.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:45.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'7962", 'reported_seq': 9212, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:40.881571+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:40.881571+0000', 'last_peered': '2024-09-10T20:25:40.881571+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:40.881571+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7962, 'log_dups_size': 0, 'ondisk_log_size': 7962, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34100384, 'num_objects': 4163, 'num_object_clones': 0, 'num_object_copies': 16652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1396, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7960, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 587, 'num_bytes_recovered': 4521984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:45.750 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:46.412 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22228/268432 objects degraded (8.281%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":435,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67108,"data_bytes":549712904,"bytes_used":2751107072,"bytes_avail":383795949568,"bytes_total":386547056640,"degraded_objects":22228,"degraded_total":268432,"degraded_ratio":0.082806818859152406,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":138528,"recovering_keys_per_sec":0,"num_objects_recovered":272,"num_bytes_recovered":1662976,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":222},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:46.425 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:46.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:47.060 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:47.060 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:47.083 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-10T20:25:46.009273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":546550792,"num_objects":66722,"num_object_clones":0,"num_object_copies":266888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66722,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107623,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1547,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127749,"ondisk_log_size":127749,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672288,"kb_used_data":1066112,"kb_used_omap":14,"kb_used_meta":1606065,"kb_avail":374815072,"statfs":{"total":386547056640,"available":383810633728,"internally_reserved":0,"allocated":1091698688,"data_stored":1090846444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1644611126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,9,2,11,12,4,0,3,0,0,1,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22462464,"num_objects":-2742,"num_object_clones":0,"num_object_copies":-10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":359,"num_bytes_recovered":1769472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004349"},"pg_stats":[{"pgid":"1.f","version":"40'7936","reported_seq":8211,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.649759+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:41.649759+0000","last_peered":"2024-09-10T20:25:41.649759+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:41.649759+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34291712,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7936,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8076","reported_seq":8205,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:44.363548+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:44.363548+0000","last_peered":"2024-09-10T20:25:44.363548+0000","last_clean":"2024-09-10T20:25:44.363548+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:44.363548+0000","last_undegraded":"2024-09-10T20:25:44.363548+0000","last_fullsized":"2024-09-10T20:25:44.363548+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":4176,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4517,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'7985","reported_seq":8181,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.586734+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:43.586734+0000","last_peered":"2024-09-10T20:25:43.586734+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:43.586734+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34036328,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7984,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2245}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8166","reported_seq":12335,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:44.488087+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:44.488087+0000","last_peered":"2024-09-10T20:25:44.488087+0000","last_clean":"2024-09-10T20:25:44.488087+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:44.488087+0000","last_undegraded":"2024-09-10T20:25:44.488087+0000","last_fullsized":"2024-09-10T20:25:44.488087+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8166,"log_dups_size":0,"ondisk_log_size":8166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'7960","reported_seq":8256,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.651448+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:41.651448+0000","last_peered":"2024-09-10T20:25:41.651448+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:41.651448+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34477216,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7906","reported_seq":8119,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.556138+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:41.556138+0000","last_peered":"2024-09-10T20:25:41.556138+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:41.556138+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7961","reported_seq":8158,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.840875+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:45.840875+0000","last_peered":"2024-09-10T20:25:45.840875+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:45.840875+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33658016,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2213}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8160","reported_seq":8326,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.793736+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:45.793736+0000","last_peered":"2024-09-10T20:25:45.793736+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:45.793736+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854624,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7843","reported_seq":11814,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:41.463728+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:41.463728+0000","last_peered":"2024-09-10T20:25:41.463728+0000","last_clean":"2024-09-10T20:25:41.463728+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:41.463728+0000","last_undegraded":"2024-09-10T20:25:41.463728+0000","last_fullsized":"2024-09-10T20:25:41.463728+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7919","reported_seq":11889,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:43.469626+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:43.469626+0000","last_peered":"2024-09-10T20:25:43.469626+0000","last_clean":"2024-09-10T20:25:43.469626+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:43.469626+0000","last_undegraded":"2024-09-10T20:25:43.469626+0000","last_fullsized":"2024-09-10T20:25:43.469626+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7919,"log_dups_size":0,"ondisk_log_size":7919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34594816,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":557,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7927","reported_seq":8191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.492790+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:41.492790+0000","last_peered":"2024-09-10T20:25:41.492790+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:41.492790+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7927,"log_dups_size":0,"ondisk_log_size":7927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34287344,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'7980","reported_seq":8271,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.684140+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:41.684140+0000","last_peered":"2024-09-10T20:25:41.684140+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:41.684140+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7980,"log_dups_size":0,"ondisk_log_size":7980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34282064,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2322}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7810","reported_seq":7987,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.428083+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:43.428083+0000","last_peered":"2024-09-10T20:25:43.428083+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:43.428083+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7810,"log_dups_size":0,"ondisk_log_size":7810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32938576,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7809,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'7952","reported_seq":8212,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:41.608991+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:41.608991+0000","last_peered":"2024-09-10T20:25:41.608991+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:41.608991+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7952,"log_dups_size":0,"ondisk_log_size":7952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33928352,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7950,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8143","reported_seq":8342,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:43.428058+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:43.428058+0000","last_peered":"2024-09-10T20:25:43.428058+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:43.428058+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8143,"log_dups_size":0,"ondisk_log_size":8143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271840,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2395}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8025","reported_seq":9465,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.891163+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:45.891163+0000","last_peered":"2024-09-10T20:25:45.891163+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:45.891163+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8025,"log_dups_size":0,"ondisk_log_size":8025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33584288,"num_objects":4100,"num_object_clones":0,"num_object_copies":16400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8023,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":546550792,"num_objects":66722,"num_object_clones":0,"num_object_copies":266888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66722,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107623,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1547,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089032192,"data_stored":1089032192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127749,"ondisk_log_size":127749,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706228,"kb_used_data":269144,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93665612,"statfs":{"total":96636764160,"available":95913586688,"internally_reserved":0,"allocated":275603456,"data_stored":275390395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,1,1,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691384,"kb_used_data":267292,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93680456,"statfs":{"total":96636764160,"available":95928786944,"internally_reserved":0,"allocated":273707008,"data_stored":273493947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,4,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585028,"kb_used_data":264564,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93786812,"statfs":{"total":96636764160,"available":96037695488,"internally_reserved":0,"allocated":270913536,"data_stored":270700475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328134354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689648,"kb_used_data":265112,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93682192,"statfs":{"total":96636764160,"available":95930564608,"internally_reserved":0,"allocated":271474688,"data_stored":271261627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270831616,"data_stored":270831616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270348288,"data_stored":270348288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272904192,"data_stored":272904192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274948096,"data_stored":274948096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:50.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:50.708 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:50.740 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-10T20:25:50.011706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539104264,"num_objects":65813,"num_object_clones":0,"num_object_copies":263252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108532,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128658,"ondisk_log_size":128658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2679028,"kb_used_data":1054812,"kb_used_omap":14,"kb_used_meta":1624113,"kb_avail":374808332,"statfs":{"total":386547056640,"available":383803731968,"internally_reserved":0,"allocated":1080127488,"data_stored":1079275244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1663092278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,8,9,18,4,7,0,0,0,3,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22773760,"num_objects":-2780,"num_object_clones":0,"num_object_copies":-11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005427"},"pg_stats":[{"pgid":"1.f","version":"40'8015","reported_seq":8292,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.576057+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:46.576057+0000","last_peered":"2024-09-10T20:25:46.576057+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:46.576057+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33644544,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8145","reported_seq":8276,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:49.524666+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:49.524666+0000","last_peered":"2024-09-10T20:25:49.524666+0000","last_clean":"2024-09-10T20:25:49.524666+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:49.524666+0000","last_undegraded":"2024-09-10T20:25:49.524666+0000","last_fullsized":"2024-09-10T20:25:49.524666+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33644544,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8043","reported_seq":8246,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.513714+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:48.513714+0000","last_peered":"2024-09-10T20:25:48.513714+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:48.513714+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33561192,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8042,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2190}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8238","reported_seq":12406,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:49.490118+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:49.490118+0000","last_peered":"2024-09-10T20:25:49.490118+0000","last_clean":"2024-09-10T20:25:49.490118+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:49.490118+0000","last_undegraded":"2024-09-10T20:25:49.490118+0000","last_fullsized":"2024-09-10T20:25:49.490118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34144256,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2498,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8047","reported_seq":8365,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.668019+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:46.668019+0000","last_peered":"2024-09-10T20:25:46.668019+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:46.668019+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33764512,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8045,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7974","reported_seq":8195,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.515409+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:46.515409+0000","last_peered":"2024-09-10T20:25:46.515409+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:46.515409+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7974,"log_dups_size":0,"ondisk_log_size":7974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33783808,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7961","reported_seq":8158,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.840875+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:45.840875+0000","last_peered":"2024-09-10T20:25:45.840875+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:45.840875+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33658016,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2213}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8160","reported_seq":8326,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.793736+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:45.793736+0000","last_peered":"2024-09-10T20:25:45.793736+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:45.793736+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854624,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7910","reported_seq":11882,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:46.590877+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:46.590877+0000","last_peered":"2024-09-10T20:25:46.590877+0000","last_clean":"2024-09-10T20:25:46.590877+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:46.590877+0000","last_undegraded":"2024-09-10T20:25:46.590877+0000","last_fullsized":"2024-09-10T20:25:46.590877+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33512016,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7994","reported_seq":11964,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:48.472483+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:48.472483+0000","last_peered":"2024-09-10T20:25:48.472483+0000","last_clean":"2024-09-10T20:25:48.472483+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:48.472483+0000","last_undegraded":"2024-09-10T20:25:48.472483+0000","last_fullsized":"2024-09-10T20:25:48.472483+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33980416,"num_objects":4148,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7995","reported_seq":8265,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.655291+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:46.655291+0000","last_peered":"2024-09-10T20:25:46.655291+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:46.655291+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33730288,"num_objects":4118,"num_object_clones":0,"num_object_copies":16472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8048","reported_seq":8344,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.683826+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:46.683826+0000","last_peered":"2024-09-10T20:25:46.683826+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:46.683826+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8048,"log_dups_size":0,"ondisk_log_size":8048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33725008,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8047,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7884","reported_seq":8067,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.536459+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:48.536459+0000","last_peered":"2024-09-10T20:25:48.536459+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:48.536459+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32332368,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8011","reported_seq":8277,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.654372+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:46.654372+0000","last_peered":"2024-09-10T20:25:46.654372+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:46.654372+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33445024,"num_objects":4083,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8208","reported_seq":8413,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.528613+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:48.528613+0000","last_peered":"2024-09-10T20:25:48.528613+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:48.528613+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739360,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8025","reported_seq":9465,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.891163+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:45.891163+0000","last_peered":"2024-09-10T20:25:45.891163+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:45.891163+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8025,"log_dups_size":0,"ondisk_log_size":8025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33584288,"num_objects":4100,"num_object_clones":0,"num_object_copies":16400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8023,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539104264,"num_objects":65813,"num_object_clones":0,"num_object_copies":263252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108532,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077084160,"data_stored":1077084160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128658,"ondisk_log_size":128658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689068,"kb_used_data":264476,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93682772,"statfs":{"total":96636764160,"available":95931158528,"internally_reserved":0,"allocated":270823424,"data_stored":270610363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,5,9,2,7,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708956,"kb_used_data":262668,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93662884,"statfs":{"total":96636764160,"available":95910793216,"internally_reserved":0,"allocated":268972032,"data_stored":268758971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591356,"kb_used_data":262556,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93780484,"statfs":{"total":96636764160,"available":96031215616,"internally_reserved":0,"allocated":268857344,"data_stored":268644283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689648,"kb_used_data":265112,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93682192,"statfs":{"total":96636764160,"available":95930564608,"internally_reserved":0,"allocated":271474688,"data_stored":271261627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270831616,"data_stored":270831616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267710464,"data_stored":267710464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268308480,"data_stored":268308480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270233600,"data_stored":270233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:50.741 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:51.329 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:51.341 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-10T20:25:50.011706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539104264,"num_objects":65813,"num_object_clones":0,"num_object_copies":263252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108532,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128658,"ondisk_log_size":128658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2679028,"kb_used_data":1054812,"kb_used_omap":14,"kb_used_meta":1624113,"kb_avail":374808332,"statfs":{"total":386547056640,"available":383803731968,"internally_reserved":0,"allocated":1080127488,"data_stored":1079275244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1663092278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,8,9,18,4,7,0,0,0,3,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22773760,"num_objects":-2780,"num_object_clones":0,"num_object_copies":-11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005427"},"pg_stats":[{"pgid":"1.f","version":"40'8015","reported_seq":8292,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.576057+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:46.576057+0000","last_peered":"2024-09-10T20:25:46.576057+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:46.576057+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33644544,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8145","reported_seq":8276,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:49.524666+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:49.524666+0000","last_peered":"2024-09-10T20:25:49.524666+0000","last_clean":"2024-09-10T20:25:49.524666+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:49.524666+0000","last_undegraded":"2024-09-10T20:25:49.524666+0000","last_fullsized":"2024-09-10T20:25:49.524666+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33644544,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8043","reported_seq":8246,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.513714+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:48.513714+0000","last_peered":"2024-09-10T20:25:48.513714+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:48.513714+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33561192,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8042,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2190}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8238","reported_seq":12406,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:49.490118+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:49.490118+0000","last_peered":"2024-09-10T20:25:49.490118+0000","last_clean":"2024-09-10T20:25:49.490118+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:49.490118+0000","last_undegraded":"2024-09-10T20:25:49.490118+0000","last_fullsized":"2024-09-10T20:25:49.490118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34144256,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2498,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8047","reported_seq":8365,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.668019+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:46.668019+0000","last_peered":"2024-09-10T20:25:46.668019+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:46.668019+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33764512,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8045,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'7974","reported_seq":8195,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.515409+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:46.515409+0000","last_peered":"2024-09-10T20:25:46.515409+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:46.515409+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7974,"log_dups_size":0,"ondisk_log_size":7974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33783808,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'7961","reported_seq":8158,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.840875+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:45.840875+0000","last_peered":"2024-09-10T20:25:45.840875+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:45.840875+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33658016,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2213}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8160","reported_seq":8326,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.793736+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:45.793736+0000","last_peered":"2024-09-10T20:25:45.793736+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:45.793736+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854624,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7910","reported_seq":11882,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:46.590877+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:46.590877+0000","last_peered":"2024-09-10T20:25:46.590877+0000","last_clean":"2024-09-10T20:25:46.590877+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:46.590877+0000","last_undegraded":"2024-09-10T20:25:46.590877+0000","last_fullsized":"2024-09-10T20:25:46.590877+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33512016,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7994","reported_seq":11964,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:48.472483+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:48.472483+0000","last_peered":"2024-09-10T20:25:48.472483+0000","last_clean":"2024-09-10T20:25:48.472483+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:48.472483+0000","last_undegraded":"2024-09-10T20:25:48.472483+0000","last_fullsized":"2024-09-10T20:25:48.472483+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33980416,"num_objects":4148,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'7995","reported_seq":8265,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.655291+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:46.655291+0000","last_peered":"2024-09-10T20:25:46.655291+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:46.655291+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33730288,"num_objects":4118,"num_object_clones":0,"num_object_copies":16472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8048","reported_seq":8344,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.683826+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:46.683826+0000","last_peered":"2024-09-10T20:25:46.683826+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:46.683826+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8048,"log_dups_size":0,"ondisk_log_size":8048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33725008,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8047,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7884","reported_seq":8067,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.536459+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:48.536459+0000","last_peered":"2024-09-10T20:25:48.536459+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:48.536459+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32332368,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8011","reported_seq":8277,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:46.654372+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:46.654372+0000","last_peered":"2024-09-10T20:25:46.654372+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:46.654372+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33445024,"num_objects":4083,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8208","reported_seq":8413,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.528613+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:48.528613+0000","last_peered":"2024-09-10T20:25:48.528613+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:48.528613+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739360,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8025","reported_seq":9465,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:45.891163+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:45.891163+0000","last_peered":"2024-09-10T20:25:45.891163+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:45.891163+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8025,"log_dups_size":0,"ondisk_log_size":8025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33584288,"num_objects":4100,"num_object_clones":0,"num_object_copies":16400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8023,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539104264,"num_objects":65813,"num_object_clones":0,"num_object_copies":263252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108532,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077084160,"data_stored":1077084160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128658,"ondisk_log_size":128658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689068,"kb_used_data":264476,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93682772,"statfs":{"total":96636764160,"available":95931158528,"internally_reserved":0,"allocated":270823424,"data_stored":270610363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,5,9,2,7,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708956,"kb_used_data":262668,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93662884,"statfs":{"total":96636764160,"available":95910793216,"internally_reserved":0,"allocated":268972032,"data_stored":268758971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591356,"kb_used_data":262556,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93780484,"statfs":{"total":96636764160,"available":96031215616,"internally_reserved":0,"allocated":268857344,"data_stored":268644283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689648,"kb_used_data":265112,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93682192,"statfs":{"total":96636764160,"available":95930564608,"internally_reserved":0,"allocated":271474688,"data_stored":271261627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270831616,"data_stored":270831616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267710464,"data_stored":267710464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268308480,"data_stored":268308480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270233600,"data_stored":270233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:51.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8015", 'reported_seq': 8292, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:46.576057+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:46.576057+0000', 'last_peered': '2024-09-10T20:25:46.576057+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:46.576057+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8015, 'log_dups_size': 0, 'ondisk_log_size': 8015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33644544, 'num_objects': 4107, 'num_object_clones': 0, 'num_object_copies': 16428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8015, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 109, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2250}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8043", 'reported_seq': 8246, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:48.513714+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:48.513714+0000', 'last_peered': '2024-09-10T20:25:48.513714+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:48.513714+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8043, 'log_dups_size': 0, 'ondisk_log_size': 8043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33561192, 'num_objects': 4098, 'num_object_clones': 0, 'num_object_copies': 16392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4098, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8042, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2190}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8047", 'reported_seq': 8365, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:46.668019+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:46.668019+0000', 'last_peered': '2024-09-10T20:25:46.668019+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:46.668019+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8047, 'log_dups_size': 0, 'ondisk_log_size': 8047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33764512, 'num_objects': 4122, 'num_object_clones': 0, 'num_object_copies': 16488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1858, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8045, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 126, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2266}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'7974", 'reported_seq': 8195, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:46.515409+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:46.515409+0000', 'last_peered': '2024-09-10T20:25:46.515409+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:46.515409+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7974, 'log_dups_size': 0, 'ondisk_log_size': 7974, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33783808, 'num_objects': 4124, 'num_object_clones': 0, 'num_object_copies': 16496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7974, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2267}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:51.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'7961", 'reported_seq': 8158, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:45.840875+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:45.840875+0000', 'last_peered': '2024-09-10T20:25:45.840875+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:45.840875+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7961, 'log_dups_size': 0, 'ondisk_log_size': 7961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33658016, 'num_objects': 4109, 'num_object_clones': 0, 'num_object_copies': 16436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1898, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7959, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2213}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8160", 'reported_seq': 8326, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:45.793736+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:45.793736+0000', 'last_peered': '2024-09-10T20:25:45.793736+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:45.793736+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8160, 'log_dups_size': 0, 'ondisk_log_size': 8160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33854624, 'num_objects': 4133, 'num_object_clones': 0, 'num_object_copies': 16532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2003, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8158, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2132}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'7995", 'reported_seq': 8265, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:46.655291+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:46.655291+0000', 'last_peered': '2024-09-10T20:25:46.655291+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:46.655291+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7995, 'log_dups_size': 0, 'ondisk_log_size': 7995, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33730288, 'num_objects': 4118, 'num_object_clones': 0, 'num_object_copies': 16472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7992, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2262}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8048", 'reported_seq': 8344, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:46.683826+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:46.683826+0000', 'last_peered': '2024-09-10T20:25:46.683826+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:46.683826+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8048, 'log_dups_size': 0, 'ondisk_log_size': 8048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33725008, 'num_objects': 4117, 'num_object_clones': 0, 'num_object_copies': 16468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8047, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:51.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:51.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7884", 'reported_seq': 8067, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:48.536459+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:48.536459+0000', 'last_peered': '2024-09-10T20:25:48.536459+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:48.536459+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7884, 'log_dups_size': 0, 'ondisk_log_size': 7884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32332368, 'num_objects': 3947, 'num_object_clones': 0, 'num_object_copies': 15788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7883, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:51.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:51.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8011", 'reported_seq': 8277, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:46.654372+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:46.654372+0000', 'last_peered': '2024-09-10T20:25:46.654372+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:46.654372+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8011, 'log_dups_size': 0, 'ondisk_log_size': 8011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33445024, 'num_objects': 4083, 'num_object_clones': 0, 'num_object_copies': 16332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1878, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8009, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2207}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:51.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:51.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8208", 'reported_seq': 8413, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:48.528613+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:48.528613+0000', 'last_peered': '2024-09-10T20:25:48.528613+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:48.528613+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8208, 'log_dups_size': 0, 'ondisk_log_size': 8208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34739360, 'num_objects': 4241, 'num_object_clones': 0, 'num_object_copies': 16964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8206, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:51.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:51.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8025", 'reported_seq': 9465, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:45.891163+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:45.891163+0000', 'last_peered': '2024-09-10T20:25:45.891163+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:45.891163+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8025, 'log_dups_size': 0, 'ondisk_log_size': 8025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33584288, 'num_objects': 4100, 'num_object_clones': 0, 'num_object_copies': 16400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1302, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8023, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 681, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:51.346 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:52.014 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22105/264904 objects degraded (8.345%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":440,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66226,"data_bytes":542487560,"bytes_used":2718851072,"bytes_avail":383828205568,"bytes_total":386547056640,"degraded_objects":22105,"degraded_total":264904,"degraded_ratio":0.083445323588922773,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":71653,"recovering_keys_per_sec":0,"num_objects_recovered":260,"num_bytes_recovered":860160,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":237},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:52.037 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:52.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:52.630 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:52.630 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:52.643 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-10T20:25:52.012853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":533132296,"num_objects":65084,"num_object_clones":0,"num_object_copies":260336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109261,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129387,"ondisk_log_size":129387,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2714216,"kb_used_data":1045588,"kb_used_omap":14,"kb_used_meta":1668529,"kb_avail":374773144,"statfs":{"total":386547056640,"available":383767699456,"internally_reserved":0,"allocated":1070682112,"data_stored":1069829868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1708574262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,12,3,5,2,3,11,0,0,2,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25157632,"num_objects":-3071,"num_object_clones":0,"num_object_copies":-12284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3071,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005718"},"pg_stats":[{"pgid":"1.f","version":"40'8090","reported_seq":8371,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.615157+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:51.615157+0000","last_peered":"2024-09-10T20:25:51.615157+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:51.615157+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33030144,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8145","reported_seq":8276,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:49.524666+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:49.524666+0000","last_peered":"2024-09-10T20:25:49.524666+0000","last_clean":"2024-09-10T20:25:49.524666+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:49.524666+0000","last_undegraded":"2024-09-10T20:25:49.524666+0000","last_fullsized":"2024-09-10T20:25:49.524666+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33644544,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8043","reported_seq":8246,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.513714+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:48.513714+0000","last_peered":"2024-09-10T20:25:48.513714+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:48.513714+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33561192,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8042,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2190}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8238","reported_seq":12406,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:49.490118+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:49.490118+0000","last_peered":"2024-09-10T20:25:49.490118+0000","last_clean":"2024-09-10T20:25:49.490118+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:49.490118+0000","last_undegraded":"2024-09-10T20:25:49.490118+0000","last_fullsized":"2024-09-10T20:25:49.490118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34144256,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2498,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8128","reported_seq":8448,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.452728+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:51.452728+0000","last_peered":"2024-09-10T20:25:51.452728+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:51.452728+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8128,"log_dups_size":0,"ondisk_log_size":8128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8045","reported_seq":8266,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.728257+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:51.728257+0000","last_peered":"2024-09-10T20:25:51.728257+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:51.728257+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8045,"log_dups_size":0,"ondisk_log_size":8045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33202176,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8045,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8040","reported_seq":8241,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:50.931904+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:50.931904+0000","last_peered":"2024-09-10T20:25:50.931904+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:50.931904+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8040,"log_dups_size":0,"ondisk_log_size":8040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33010848,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2137}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8225","reported_seq":8393,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:50.666049+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:50.666049+0000","last_peered":"2024-09-10T20:25:50.666049+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:50.666049+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8225,"log_dups_size":0,"ondisk_log_size":8225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33322144,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8223,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7987","reported_seq":11958,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:51.602647+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:51.602647+0000","last_peered":"2024-09-10T20:25:51.602647+0000","last_clean":"2024-09-10T20:25:51.602647+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:51.602647+0000","last_undegraded":"2024-09-10T20:25:51.602647+0000","last_fullsized":"2024-09-10T20:25:51.602647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32881232,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4540,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'7994","reported_seq":11964,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:48.472483+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:48.472483+0000","last_peered":"2024-09-10T20:25:48.472483+0000","last_clean":"2024-09-10T20:25:48.472483+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:48.472483+0000","last_undegraded":"2024-09-10T20:25:48.472483+0000","last_fullsized":"2024-09-10T20:25:48.472483+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33980416,"num_objects":4148,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8067","reported_seq":8339,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.715639+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:51.715639+0000","last_peered":"2024-09-10T20:25:51.715639+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:51.715639+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33140464,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8064,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8127","reported_seq":8422,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.685963+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:51.685963+0000","last_peered":"2024-09-10T20:25:51.685963+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:51.685963+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33077840,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7884","reported_seq":8067,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.536459+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:48.536459+0000","last_peered":"2024-09-10T20:25:48.536459+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:48.536459+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32332368,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8082","reported_seq":8349,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.713912+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:51.713912+0000","last_peered":"2024-09-10T20:25:51.713912+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:51.713912+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863392,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2137}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8208","reported_seq":8413,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:48.528613+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:48.528613+0000","last_peered":"2024-09-10T20:25:48.528613+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:48.528613+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739360,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8084","reported_seq":9804,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.045221+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:51.045221+0000","last_peered":"2024-09-10T20:25:51.045221+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:51.045221+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8084,"log_dups_size":0,"ondisk_log_size":8084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":821,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":533132296,"num_objects":65084,"num_object_clones":0,"num_object_copies":260336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109261,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1067438080,"data_stored":1067438080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129387,"ondisk_log_size":129387,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706656,"kb_used_data":259852,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93665184,"statfs":{"total":96636764160,"available":95913148416,"internally_reserved":0,"allocated":266088448,"data_stored":265875387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":457502416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708956,"kb_used_data":262668,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93662884,"statfs":{"total":96636764160,"available":95910793216,"internally_reserved":0,"allocated":268972032,"data_stored":268758971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591356,"kb_used_data":262556,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93780484,"statfs":{"total":96636764160,"available":96031215616,"internally_reserved":0,"allocated":268857344,"data_stored":268644283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707248,"kb_used_data":260512,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93664592,"statfs":{"total":96636764160,"available":95912542208,"internally_reserved":0,"allocated":266764288,"data_stored":266551227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457436879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10,1,0,2,1,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265998336,"data_stored":265998336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267710464,"data_stored":267710464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268308480,"data_stored":268308480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265420800,"data_stored":265420800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:55.645 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:56.254 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:56.255 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-10T20:25:56.014342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":529617928,"num_objects":64655,"num_object_clones":0,"num_object_copies":258620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109690,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129816,"ondisk_log_size":129816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2696824,"kb_used_data":1036524,"kb_used_omap":14,"kb_used_meta":1660209,"kb_avail":374790536,"statfs":{"total":386547056640,"available":383785508864,"internally_reserved":0,"allocated":1061400576,"data_stored":1060548332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1700054582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,12,12,9,8,6,14,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20094976,"num_objects":-2453,"num_object_clones":0,"num_object_copies":-9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":106496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005721"},"pg_stats":[{"pgid":"1.f","version":"40'8090","reported_seq":8371,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.615157+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:51.615157+0000","last_peered":"2024-09-10T20:25:51.615157+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:51.615157+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33030144,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8213","reported_seq":8344,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:54.619615+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:54.619615+0000","last_peered":"2024-09-10T20:25:54.619615+0000","last_clean":"2024-09-10T20:25:54.619615+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:54.619615+0000","last_undegraded":"2024-09-10T20:25:54.619615+0000","last_fullsized":"2024-09-10T20:25:54.619615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8213,"log_dups_size":0,"ondisk_log_size":8213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33087488,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8125","reported_seq":8327,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.588639+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:53.588639+0000","last_peered":"2024-09-10T20:25:53.588639+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:53.588639+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32889448,"num_objects":4016,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4016,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8124,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8318","reported_seq":12487,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:54.600118+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:54.600118+0000","last_peered":"2024-09-10T20:25:54.600118+0000","last_clean":"2024-09-10T20:25:54.600118+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:54.600118+0000","last_undegraded":"2024-09-10T20:25:54.600118+0000","last_fullsized":"2024-09-10T20:25:54.600118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":4088,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8128","reported_seq":8448,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.452728+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:51.452728+0000","last_peered":"2024-09-10T20:25:51.452728+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:51.452728+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8128,"log_dups_size":0,"ondisk_log_size":8128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8045","reported_seq":8266,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.728257+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:51.728257+0000","last_peered":"2024-09-10T20:25:51.728257+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:51.728257+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8045,"log_dups_size":0,"ondisk_log_size":8045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33202176,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8045,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8040","reported_seq":8241,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:50.931904+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:50.931904+0000","last_peered":"2024-09-10T20:25:50.931904+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:50.931904+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8040,"log_dups_size":0,"ondisk_log_size":8040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33010848,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2137}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8225","reported_seq":8393,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:50.666049+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:50.666049+0000","last_peered":"2024-09-10T20:25:50.666049+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:50.666049+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8225,"log_dups_size":0,"ondisk_log_size":8225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33322144,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8223,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7987","reported_seq":11958,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:51.602647+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:51.602647+0000","last_peered":"2024-09-10T20:25:51.602647+0000","last_clean":"2024-09-10T20:25:51.602647+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:51.602647+0000","last_undegraded":"2024-09-10T20:25:51.602647+0000","last_fullsized":"2024-09-10T20:25:51.602647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32881232,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4540,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8066","reported_seq":12036,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:53.578415+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:53.578415+0000","last_peered":"2024-09-10T20:25:53.578415+0000","last_clean":"2024-09-10T20:25:53.578415+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:53.578415+0000","last_undegraded":"2024-09-10T20:25:53.578415+0000","last_fullsized":"2024-09-10T20:25:53.578415+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8066,"log_dups_size":0,"ondisk_log_size":8066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":4076,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8067","reported_seq":8339,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.715639+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:51.715639+0000","last_peered":"2024-09-10T20:25:51.715639+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:51.715639+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33140464,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8064,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8127","reported_seq":8422,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.685963+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:51.685963+0000","last_peered":"2024-09-10T20:25:51.685963+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:51.685963+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33077840,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7935","reported_seq":8118,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.342950+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:53.342950+0000","last_peered":"2024-09-10T20:25:53.342950+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:53.342950+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31914576,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7934,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8082","reported_seq":8349,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.713912+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:51.713912+0000","last_peered":"2024-09-10T20:25:51.713912+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:51.713912+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863392,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2137}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8284","reported_seq":8489,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.525367+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:53.525367+0000","last_peered":"2024-09-10T20:25:53.525367+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:53.525367+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34116768,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8084","reported_seq":9804,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.045221+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:51.045221+0000","last_peered":"2024-09-10T20:25:51.045221+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:51.045221+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8084,"log_dups_size":0,"ondisk_log_size":8084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":821,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":529617928,"num_objects":64655,"num_object_clones":0,"num_object_copies":258620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109690,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057935360,"data_stored":1057935360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129816,"ondisk_log_size":129816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706656,"kb_used_data":259852,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93665184,"statfs":{"total":96636764160,"available":95913148416,"internally_reserved":0,"allocated":266088448,"data_stored":265875387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":457502416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708684,"kb_used_data":258232,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93663156,"statfs":{"total":96636764160,"available":95911071744,"internally_reserved":0,"allocated":264429568,"data_stored":264216507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574236,"kb_used_data":257928,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93797604,"statfs":{"total":96636764160,"available":96048746496,"internally_reserved":0,"allocated":264118272,"data_stored":263905211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707248,"kb_used_data":260512,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93664592,"statfs":{"total":96636764160,"available":95912542208,"internally_reserved":0,"allocated":266764288,"data_stored":266551227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457436879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10,1,0,2,1,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265998336,"data_stored":265998336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262868992,"data_stored":262868992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263647232,"data_stored":263647232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265420800,"data_stored":265420800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:56.286 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:56.877 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:56.910 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-10T20:25:56.014342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":529617928,"num_objects":64655,"num_object_clones":0,"num_object_copies":258620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109690,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129816,"ondisk_log_size":129816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2696824,"kb_used_data":1036524,"kb_used_omap":14,"kb_used_meta":1660209,"kb_avail":374790536,"statfs":{"total":386547056640,"available":383785508864,"internally_reserved":0,"allocated":1061400576,"data_stored":1060548332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1700054582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,12,12,9,8,6,14,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20094976,"num_objects":-2453,"num_object_clones":0,"num_object_copies":-9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":106496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005721"},"pg_stats":[{"pgid":"1.f","version":"40'8090","reported_seq":8371,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.615157+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:51.615157+0000","last_peered":"2024-09-10T20:25:51.615157+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:51.615157+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33030144,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8213","reported_seq":8344,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:54.619615+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:54.619615+0000","last_peered":"2024-09-10T20:25:54.619615+0000","last_clean":"2024-09-10T20:25:54.619615+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:54.619615+0000","last_undegraded":"2024-09-10T20:25:54.619615+0000","last_fullsized":"2024-09-10T20:25:54.619615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8213,"log_dups_size":0,"ondisk_log_size":8213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33087488,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8125","reported_seq":8327,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.588639+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:53.588639+0000","last_peered":"2024-09-10T20:25:53.588639+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:53.588639+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32889448,"num_objects":4016,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4016,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8124,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8318","reported_seq":12487,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:54.600118+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:54.600118+0000","last_peered":"2024-09-10T20:25:54.600118+0000","last_clean":"2024-09-10T20:25:54.600118+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:54.600118+0000","last_undegraded":"2024-09-10T20:25:54.600118+0000","last_fullsized":"2024-09-10T20:25:54.600118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":4088,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8128","reported_seq":8448,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.452728+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:51.452728+0000","last_peered":"2024-09-10T20:25:51.452728+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:51.452728+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8128,"log_dups_size":0,"ondisk_log_size":8128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8045","reported_seq":8266,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.728257+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:51.728257+0000","last_peered":"2024-09-10T20:25:51.728257+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:51.728257+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8045,"log_dups_size":0,"ondisk_log_size":8045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33202176,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8045,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8040","reported_seq":8241,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:50.931904+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:50.931904+0000","last_peered":"2024-09-10T20:25:50.931904+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:50.931904+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8040,"log_dups_size":0,"ondisk_log_size":8040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33010848,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2137}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8225","reported_seq":8393,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:50.666049+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:50.666049+0000","last_peered":"2024-09-10T20:25:50.666049+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:50.666049+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8225,"log_dups_size":0,"ondisk_log_size":8225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33322144,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8223,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'7987","reported_seq":11958,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:51.602647+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:51.602647+0000","last_peered":"2024-09-10T20:25:51.602647+0000","last_clean":"2024-09-10T20:25:51.602647+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:51.602647+0000","last_undegraded":"2024-09-10T20:25:51.602647+0000","last_fullsized":"2024-09-10T20:25:51.602647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32881232,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4540,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8066","reported_seq":12036,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:53.578415+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:53.578415+0000","last_peered":"2024-09-10T20:25:53.578415+0000","last_clean":"2024-09-10T20:25:53.578415+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:53.578415+0000","last_undegraded":"2024-09-10T20:25:53.578415+0000","last_fullsized":"2024-09-10T20:25:53.578415+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8066,"log_dups_size":0,"ondisk_log_size":8066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":4076,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8067","reported_seq":8339,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.715639+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:51.715639+0000","last_peered":"2024-09-10T20:25:51.715639+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:51.715639+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33140464,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8064,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8127","reported_seq":8422,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.685963+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:51.685963+0000","last_peered":"2024-09-10T20:25:51.685963+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:51.685963+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33077840,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7935","reported_seq":8118,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.342950+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:53.342950+0000","last_peered":"2024-09-10T20:25:53.342950+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:53.342950+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31914576,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7934,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8082","reported_seq":8349,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.713912+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:51.713912+0000","last_peered":"2024-09-10T20:25:51.713912+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:51.713912+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863392,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2137}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8284","reported_seq":8489,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.525367+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:53.525367+0000","last_peered":"2024-09-10T20:25:53.525367+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:53.525367+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34116768,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8084","reported_seq":9804,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:51.045221+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:51.045221+0000","last_peered":"2024-09-10T20:25:51.045221+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:51.045221+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8084,"log_dups_size":0,"ondisk_log_size":8084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":821,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":529617928,"num_objects":64655,"num_object_clones":0,"num_object_copies":258620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109690,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057935360,"data_stored":1057935360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129816,"ondisk_log_size":129816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706656,"kb_used_data":259852,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93665184,"statfs":{"total":96636764160,"available":95913148416,"internally_reserved":0,"allocated":266088448,"data_stored":265875387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":457502416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708684,"kb_used_data":258232,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93663156,"statfs":{"total":96636764160,"available":95911071744,"internally_reserved":0,"allocated":264429568,"data_stored":264216507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574236,"kb_used_data":257928,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93797604,"statfs":{"total":96636764160,"available":96048746496,"internally_reserved":0,"allocated":264118272,"data_stored":263905211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707248,"kb_used_data":260512,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93664592,"statfs":{"total":96636764160,"available":95912542208,"internally_reserved":0,"allocated":266764288,"data_stored":266551227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457436879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10,1,0,2,1,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265998336,"data_stored":265998336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262868992,"data_stored":262868992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263647232,"data_stored":263647232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265420800,"data_stored":265420800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:25:56.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:25:56.911 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8090", 'reported_seq': 8371, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:51.615157+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:25:51.615157+0000', 'last_peered': '2024-09-10T20:25:51.615157+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:25:51.615157+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8090, 'log_dups_size': 0, 'ondisk_log_size': 8090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33030144, 'num_objects': 4032, 'num_object_clones': 0, 'num_object_copies': 16128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8090, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:56.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:25:56.911 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8125", 'reported_seq': 8327, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:53.588639+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:53.588639+0000', 'last_peered': '2024-09-10T20:25:53.588639+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:53.588639+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8125, 'log_dups_size': 0, 'ondisk_log_size': 8125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32889448, 'num_objects': 4016, 'num_object_clones': 0, 'num_object_copies': 16064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4016, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8124, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2108}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:56.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:25:56.911 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8128", 'reported_seq': 8448, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:51.452728+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:25:51.452728+0000', 'last_peered': '2024-09-10T20:25:51.452728+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:25:51.452728+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8128, 'log_dups_size': 0, 'ondisk_log_size': 8128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33100960, 'num_objects': 4041, 'num_object_clones': 0, 'num_object_copies': 16164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8126, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8045", 'reported_seq': 8266, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:51.728257+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:25:51.728257+0000', 'last_peered': '2024-09-10T20:25:51.728257+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:25:51.728257+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8045, 'log_dups_size': 0, 'ondisk_log_size': 8045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33202176, 'num_objects': 4053, 'num_object_clones': 0, 'num_object_copies': 16212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8045, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8040", 'reported_seq': 8241, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:50.931904+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:25:50.931904+0000', 'last_peered': '2024-09-10T20:25:50.931904+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:25:50.931904+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8040, 'log_dups_size': 0, 'ondisk_log_size': 8040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33010848, 'num_objects': 4030, 'num_object_clones': 0, 'num_object_copies': 16120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8038, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2137}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8225", 'reported_seq': 8393, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:50.666049+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:25:50.666049+0000', 'last_peered': '2024-09-10T20:25:50.666049+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:25:50.666049+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8225, 'log_dups_size': 0, 'ondisk_log_size': 8225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33322144, 'num_objects': 4068, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8223, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2068}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8067", 'reported_seq': 8339, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:51.715639+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:25:51.715639+0000', 'last_peered': '2024-09-10T20:25:51.715639+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:25:51.715639+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8067, 'log_dups_size': 0, 'ondisk_log_size': 8067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33140464, 'num_objects': 4046, 'num_object_clones': 0, 'num_object_copies': 16184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8064, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2190}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8127", 'reported_seq': 8422, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:51.685963+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:25:51.685963+0000', 'last_peered': '2024-09-10T20:25:51.685963+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:25:51.685963+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8127, 'log_dups_size': 0, 'ondisk_log_size': 8127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33077840, 'num_objects': 4038, 'num_object_clones': 0, 'num_object_copies': 16152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8126, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:56.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:25:56.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'7935", 'reported_seq': 8118, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:53.342950+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:53.342950+0000', 'last_peered': '2024-09-10T20:25:53.342950+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:53.342950+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 7935, 'log_dups_size': 0, 'ondisk_log_size': 7935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31914576, 'num_objects': 3896, 'num_object_clones': 0, 'num_object_copies': 15584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7934, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:56.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:25:56.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8082", 'reported_seq': 8349, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:51.713912+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:25:51.713912+0000', 'last_peered': '2024-09-10T20:25:51.713912+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:25:51.713912+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8082, 'log_dups_size': 0, 'ondisk_log_size': 8082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32863392, 'num_objects': 4012, 'num_object_clones': 0, 'num_object_copies': 16048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8080, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:25:56.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:25:56.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8284", 'reported_seq': 8489, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:53.525367+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:53.525367+0000', 'last_peered': '2024-09-10T20:25:53.525367+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:53.525367+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8284, 'log_dups_size': 0, 'ondisk_log_size': 8284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34116768, 'num_objects': 4165, 'num_object_clones': 0, 'num_object_copies': 16660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8282, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:25:56.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:25:56.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8084", 'reported_seq': 9804, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:51.045221+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:25:51.045221+0000', 'last_peered': '2024-09-10T20:25:51.045221+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:25:51.045221+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8084, 'log_dups_size': 0, 'ondisk_log_size': 8084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33100960, 'num_objects': 4041, 'num_object_clones': 0, 'num_object_copies': 16164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1162, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8082, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 821, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:25:56.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:25:57.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:57.762 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21948/258620 objects degraded (8.487%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":446,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64655,"data_bytes":529617928,"bytes_used":2761547776,"bytes_avail":383785508864,"bytes_total":386547056640,"degraded_objects":21948,"degraded_total":258620,"degraded_ratio":0.084865826308870157,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":8870,"recovering_keys_per_sec":0,"num_objects_recovered":280,"num_bytes_recovered":106496,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:25:57.815 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:25:57.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:25:58.458 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:25:58.459 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:25:58.483 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-10T20:25:58.018537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523318280,"num_objects":63886,"num_object_clones":0,"num_object_copies":255544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63886,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110459,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130585,"ondisk_log_size":130585,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695636,"kb_used_data":1027008,"kb_used_omap":14,"kb_used_meta":1668529,"kb_avail":374791724,"statfs":{"total":386547056640,"available":383786725376,"internally_reserved":0,"allocated":1051656192,"data_stored":1050803948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1708574262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,17,21,12,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23232512,"num_objects":-2836,"num_object_clones":0,"num_object_copies":-11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009263"},"pg_stats":[{"pgid":"1.f","version":"40'8163","reported_seq":8444,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.573062+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:56.573062+0000","last_peered":"2024-09-10T20:25:56.573062+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:56.573062+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32432128,"num_objects":3959,"num_object_clones":0,"num_object_copies":15836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8163,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8213","reported_seq":8344,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:54.619615+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:54.619615+0000","last_peered":"2024-09-10T20:25:54.619615+0000","last_clean":"2024-09-10T20:25:54.619615+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:54.619615+0000","last_undegraded":"2024-09-10T20:25:54.619615+0000","last_fullsized":"2024-09-10T20:25:54.619615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8213,"log_dups_size":0,"ondisk_log_size":8213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33087488,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8125","reported_seq":8327,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.588639+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:53.588639+0000","last_peered":"2024-09-10T20:25:53.588639+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:53.588639+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32889448,"num_objects":4016,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4016,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8124,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8318","reported_seq":12487,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:54.600118+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:54.600118+0000","last_peered":"2024-09-10T20:25:54.600118+0000","last_clean":"2024-09-10T20:25:54.600118+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:54.600118+0000","last_undegraded":"2024-09-10T20:25:54.600118+0000","last_fullsized":"2024-09-10T20:25:54.600118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":4088,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8232","reported_seq":8552,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.713713+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:56.713713+0000","last_peered":"2024-09-10T20:25:56.713713+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:56.713713+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32248992,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8102","reported_seq":8323,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.646356+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:56.646356+0000","last_peered":"2024-09-10T20:25:56.646356+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:56.646356+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8102,"log_dups_size":0,"ondisk_log_size":8102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32735232,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8102,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8114","reported_seq":8317,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:55.975934+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:55.975934+0000","last_peered":"2024-09-10T20:25:55.975934+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:55.975934+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8114,"log_dups_size":0,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32404640,"num_objects":3956,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8293","reported_seq":8461,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:55.960716+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:55.960716+0000","last_peered":"2024-09-10T20:25:55.960716+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:55.960716+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32765088,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8291,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8068","reported_seq":12040,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:56.691394+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:56.691394+0000","last_peered":"2024-09-10T20:25:56.691394+0000","last_clean":"2024-09-10T20:25:56.691394+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:56.691394+0000","last_undegraded":"2024-09-10T20:25:56.691394+0000","last_fullsized":"2024-09-10T20:25:56.691394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8068,"log_dups_size":0,"ondisk_log_size":8068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32217680,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8066","reported_seq":12036,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:53.578415+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:53.578415+0000","last_peered":"2024-09-10T20:25:53.578415+0000","last_clean":"2024-09-10T20:25:53.578415+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:53.578415+0000","last_undegraded":"2024-09-10T20:25:53.578415+0000","last_fullsized":"2024-09-10T20:25:53.578415+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8066,"log_dups_size":0,"ondisk_log_size":8066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":4076,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8145","reported_seq":8417,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.755771+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:56.755771+0000","last_peered":"2024-09-10T20:25:56.755771+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:56.755771+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32501488,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8202","reported_seq":8499,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.268484+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:56.268484+0000","last_peered":"2024-09-10T20:25:56.268484+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:56.268484+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32463440,"num_objects":3963,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8201,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'7935","reported_seq":8118,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.342950+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:53.342950+0000","last_peered":"2024-09-10T20:25:53.342950+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:53.342950+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31914576,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7934,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8172","reported_seq":8440,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.821226+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:56.821226+0000","last_peered":"2024-09-10T20:25:56.821226+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:56.821226+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8172,"log_dups_size":0,"ondisk_log_size":8172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32126112,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8170,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2047}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8284","reported_seq":8489,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:53.525367+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:53.525367+0000","last_peered":"2024-09-10T20:25:53.525367+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:53.525367+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34116768,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8153","reported_seq":10121,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.033613+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:56.033613+0000","last_peered":"2024-09-10T20:25:56.033613+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:56.033613+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535712,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":945,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":523318280,"num_objects":63886,"num_object_clones":0,"num_object_copies":255544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63886,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110459,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048080384,"data_stored":1048080384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130585,"ondisk_log_size":130585,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117074,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705932,"kb_used_data":254968,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93665908,"statfs":{"total":96636764160,"available":95913889792,"internally_reserved":0,"allocated":261087232,"data_stored":260874171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":461762256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,7,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708684,"kb_used_data":258232,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93663156,"statfs":{"total":96636764160,"available":95911071744,"internally_reserved":0,"allocated":264429568,"data_stored":264216507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574236,"kb_used_data":257928,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93797604,"statfs":{"total":96636764160,"available":96048746496,"internally_reserved":0,"allocated":264118272,"data_stored":263905211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706784,"kb_used_data":255880,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93665056,"statfs":{"total":96636764160,"available":95913017344,"internally_reserved":0,"allocated":262021120,"data_stored":261808059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":461696719},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261132288,"data_stored":261132288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262868992,"data_stored":262868992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263647232,"data_stored":263647232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260431872,"data_stored":260431872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:01.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:02.022 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:02.023 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:02.050 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-10T20:26:00.019447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519681032,"num_objects":63442,"num_object_clones":0,"num_object_copies":253768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110903,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131029,"ondisk_log_size":131029,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766452,"kb_used_data":1017284,"kb_used_omap":14,"kb_used_meta":1749041,"kb_avail":374720908,"statfs":{"total":386547056640,"available":383714209792,"internally_reserved":0,"allocated":1041698816,"data_stored":1040846572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1791018550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,4,6,18,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22806528,"num_objects":-2784,"num_object_clones":0,"num_object_copies":-11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009317"},"pg_stats":[{"pgid":"1.f","version":"40'8163","reported_seq":8444,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.573062+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:25:56.573062+0000","last_peered":"2024-09-10T20:25:56.573062+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:25:56.573062+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32432128,"num_objects":3959,"num_object_clones":0,"num_object_copies":15836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8163,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8286","reported_seq":8417,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:59.585750+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:59.585750+0000","last_peered":"2024-09-10T20:25:59.585750+0000","last_clean":"2024-09-10T20:25:59.585750+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:59.585750+0000","last_undegraded":"2024-09-10T20:25:59.585750+0000","last_fullsized":"2024-09-10T20:25:59.585750+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8286,"log_dups_size":0,"ondisk_log_size":8286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32489472,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8200","reported_seq":8403,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.401912+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:58.401912+0000","last_peered":"2024-09-10T20:25:58.401912+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:58.401912+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275048,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8199,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8381","reported_seq":12550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:59.419664+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:59.419664+0000","last_peered":"2024-09-10T20:25:59.419664+0000","last_clean":"2024-09-10T20:25:59.419664+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:59.419664+0000","last_undegraded":"2024-09-10T20:25:59.419664+0000","last_fullsized":"2024-09-10T20:25:59.419664+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32972800,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2641,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8232","reported_seq":8552,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.713713+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:25:56.713713+0000","last_peered":"2024-09-10T20:25:56.713713+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:25:56.713713+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32248992,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8102","reported_seq":8323,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.646356+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:25:56.646356+0000","last_peered":"2024-09-10T20:25:56.646356+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:25:56.646356+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8102,"log_dups_size":0,"ondisk_log_size":8102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32735232,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8102,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8114","reported_seq":8317,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:55.975934+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:25:55.975934+0000","last_peered":"2024-09-10T20:25:55.975934+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:25:55.975934+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8114,"log_dups_size":0,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32404640,"num_objects":3956,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8293","reported_seq":8461,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:55.960716+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:25:55.960716+0000","last_peered":"2024-09-10T20:25:55.960716+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:25:55.960716+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32765088,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8291,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8068","reported_seq":12040,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:56.691394+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:25:56.691394+0000","last_peered":"2024-09-10T20:25:56.691394+0000","last_clean":"2024-09-10T20:25:56.691394+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:25:56.691394+0000","last_undegraded":"2024-09-10T20:25:56.691394+0000","last_fullsized":"2024-09-10T20:25:56.691394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8068,"log_dups_size":0,"ondisk_log_size":8068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32217680,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8142","reported_seq":12112,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:58.603336+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:58.603336+0000","last_peered":"2024-09-10T20:25:58.603336+0000","last_clean":"2024-09-10T20:25:58.603336+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:58.603336+0000","last_undegraded":"2024-09-10T20:25:58.603336+0000","last_fullsized":"2024-09-10T20:25:58.603336+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8145","reported_seq":8417,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.755771+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:25:56.755771+0000","last_peered":"2024-09-10T20:25:56.755771+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:25:56.755771+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32501488,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8202","reported_seq":8499,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.268484+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:25:56.268484+0000","last_peered":"2024-09-10T20:25:56.268484+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:25:56.268484+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32463440,"num_objects":3963,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8201,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8010","reported_seq":8193,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.648121+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:58.648121+0000","last_peered":"2024-09-10T20:25:58.648121+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:58.648121+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8010,"log_dups_size":0,"ondisk_log_size":8010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300176,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8172","reported_seq":8440,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.821226+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:25:56.821226+0000","last_peered":"2024-09-10T20:25:56.821226+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:25:56.821226+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8172,"log_dups_size":0,"ondisk_log_size":8172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32126112,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8170,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2047}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8366","reported_seq":8571,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.511245+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:58.511245+0000","last_peered":"2024-09-10T20:25:58.511245+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:58.511245+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33445024,"num_objects":4083,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8153","reported_seq":10121,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:56.033613+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:25:56.033613+0000","last_peered":"2024-09-10T20:25:56.033613+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:25:56.033613+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535712,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":945,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":519681032,"num_objects":63442,"num_object_clones":0,"num_object_copies":253768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110903,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038151680,"data_stored":1038151680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131029,"ondisk_log_size":131029,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117074,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705932,"kb_used_data":254968,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93665908,"statfs":{"total":96636764160,"available":95913889792,"internally_reserved":0,"allocated":261087232,"data_stored":260874171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":461762256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,7,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726056,"kb_used_data":253368,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93645784,"statfs":{"total":96636764160,"available":95893282816,"internally_reserved":0,"allocated":259448832,"data_stored":259235771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627680,"kb_used_data":253068,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93744160,"statfs":{"total":96636764160,"available":95994019840,"internally_reserved":0,"allocated":259141632,"data_stored":258928571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":383577810},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706784,"kb_used_data":255880,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93665056,"statfs":{"total":96636764160,"available":95913017344,"internally_reserved":0,"allocated":262021120,"data_stored":261808059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":461696719},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261132288,"data_stored":261132288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257912832,"data_stored":257912832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258674688,"data_stored":258674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260431872,"data_stored":260431872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:02.051 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:02.629 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:02.655 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-10T20:26:02.020306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513274888,"num_objects":62660,"num_object_clones":0,"num_object_copies":250640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111685,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131811,"ondisk_log_size":131811,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801024,"kb_used_data":1007424,"kb_used_omap":14,"kb_used_meta":1793457,"kb_avail":374686336,"statfs":{"total":386547056640,"available":383678808064,"internally_reserved":0,"allocated":1031602176,"data_stored":1030749932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1836500534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,7,10,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25829376,"num_objects":-3153,"num_object_clones":0,"num_object_copies":-12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008599"},"pg_stats":[{"pgid":"1.f","version":"40'8247","reported_seq":8528,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.617485+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:01.617485+0000","last_peered":"2024-09-10T20:26:01.617485+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:01.617485+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8247,"log_dups_size":0,"ondisk_log_size":8247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31744000,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8286","reported_seq":8417,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:59.585750+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:59.585750+0000","last_peered":"2024-09-10T20:25:59.585750+0000","last_clean":"2024-09-10T20:25:59.585750+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:59.585750+0000","last_undegraded":"2024-09-10T20:25:59.585750+0000","last_fullsized":"2024-09-10T20:25:59.585750+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8286,"log_dups_size":0,"ondisk_log_size":8286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32489472,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8200","reported_seq":8403,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.401912+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:58.401912+0000","last_peered":"2024-09-10T20:25:58.401912+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:58.401912+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275048,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8199,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8381","reported_seq":12550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:59.419664+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:59.419664+0000","last_peered":"2024-09-10T20:25:59.419664+0000","last_clean":"2024-09-10T20:25:59.419664+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:59.419664+0000","last_undegraded":"2024-09-10T20:25:59.419664+0000","last_fullsized":"2024-09-10T20:25:59.419664+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32972800,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2641,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8314","reported_seq":8632,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.760951+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:01.760951+0000","last_peered":"2024-09-10T20:26:01.760951+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:01.760951+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8314,"log_dups_size":0,"ondisk_log_size":8314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8194","reported_seq":8415,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.708683+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:01.708683+0000","last_peered":"2024-09-10T20:26:01.708683+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:01.708683+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8194,"log_dups_size":0,"ondisk_log_size":8194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":3904,"num_object_clones":0,"num_object_copies":15616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8195","reported_seq":8398,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.074643+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:01.074643+0000","last_peered":"2024-09-10T20:26:01.074643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:01.074643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31741088,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1982}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8361","reported_seq":8529,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:00.867125+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:00.867125+0000","last_peered":"2024-09-10T20:26:00.867125+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:00.867125+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8142","reported_seq":12113,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:01.797029+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:01.797029+0000","last_peered":"2024-09-10T20:26:01.797029+0000","last_clean":"2024-09-10T20:26:01.797029+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:01.797029+0000","last_undegraded":"2024-09-10T20:26:01.797029+0000","last_fullsized":"2024-09-10T20:26:01.797029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31611472,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8142","reported_seq":12112,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:58.603336+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:58.603336+0000","last_peered":"2024-09-10T20:25:58.603336+0000","last_clean":"2024-09-10T20:25:58.603336+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:58.603336+0000","last_undegraded":"2024-09-10T20:25:58.603336+0000","last_fullsized":"2024-09-10T20:25:58.603336+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8230","reported_seq":8503,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.640882+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:01.640882+0000","last_peered":"2024-09-10T20:26:01.640882+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:01.640882+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31805168,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8278","reported_seq":8574,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.802548+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:01.802548+0000","last_peered":"2024-09-10T20:26:01.802548+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:01.802548+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31840848,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8010","reported_seq":8193,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.648121+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:58.648121+0000","last_peered":"2024-09-10T20:25:58.648121+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:58.648121+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8010,"log_dups_size":0,"ondisk_log_size":8010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300176,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8241","reported_seq":8509,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.709892+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:01.709892+0000","last_peered":"2024-09-10T20:26:01.709892+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:01.709892+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560864,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8366","reported_seq":8571,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.511245+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:58.511245+0000","last_peered":"2024-09-10T20:25:58.511245+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:58.511245+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33445024,"num_objects":4083,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8224","reported_seq":10339,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.096842+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:01.096842+0000","last_peered":"2024-09-10T20:26:01.096842+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:01.096842+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8224,"log_dups_size":0,"ondisk_log_size":8224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31954080,"num_objects":3901,"num_object_clones":0,"num_object_copies":15604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513274888,"num_objects":62660,"num_object_clones":0,"num_object_copies":250640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111685,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1028038656,"data_stored":1028038656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131811,"ondisk_log_size":131811,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723156,"kb_used_data":249988,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93648684,"statfs":{"total":96636764160,"available":95896252416,"internally_reserved":0,"allocated":255987712,"data_stored":255774651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726056,"kb_used_data":253368,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93645784,"statfs":{"total":96636764160,"available":95893282816,"internally_reserved":0,"allocated":259448832,"data_stored":259235771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627680,"kb_used_data":253068,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93744160,"statfs":{"total":96636764160,"available":95994019840,"internally_reserved":0,"allocated":259141632,"data_stored":258928571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":383577810},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724132,"kb_used_data":251000,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93647708,"statfs":{"total":96636764160,"available":95895252992,"internally_reserved":0,"allocated":257024000,"data_stored":256810939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256077824,"data_stored":256077824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257912832,"data_stored":257912832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258674688,"data_stored":258674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":255373312,"data_stored":255373312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8247", 'reported_seq': 8528, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.617485+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:01.617485+0000', 'last_peered': '2024-09-10T20:26:01.617485+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:01.617485+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8247, 'log_dups_size': 0, 'ondisk_log_size': 8247, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31744000, 'num_objects': 3875, 'num_object_clones': 0, 'num_object_copies': 15500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8247, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2020}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8200", 'reported_seq': 8403, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:58.401912+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:25:58.401912+0000', 'last_peered': '2024-09-10T20:25:58.401912+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:25:58.401912+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8200, 'log_dups_size': 0, 'ondisk_log_size': 8200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32275048, 'num_objects': 3941, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3941, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8199, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2033}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8314", 'reported_seq': 8632, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.760951+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:01.760951+0000', 'last_peered': '2024-09-10T20:26:01.760951+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:01.760951+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8314, 'log_dups_size': 0, 'ondisk_log_size': 8314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31577248, 'num_objects': 3855, 'num_object_clones': 0, 'num_object_copies': 15420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8312, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2000}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8194", 'reported_seq': 8415, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.708683+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:01.708683+0000', 'last_peered': '2024-09-10T20:26:01.708683+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:01.708683+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8194, 'log_dups_size': 0, 'ondisk_log_size': 8194, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31981568, 'num_objects': 3904, 'num_object_clones': 0, 'num_object_copies': 15616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8194, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:02.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8195", 'reported_seq': 8398, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.074643+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:01.074643+0000', 'last_peered': '2024-09-10T20:26:01.074643+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:01.074643+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8195, 'log_dups_size': 0, 'ondisk_log_size': 8195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31741088, 'num_objects': 3875, 'num_object_clones': 0, 'num_object_copies': 15500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8193, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1982}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8361", 'reported_seq': 8529, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:00.867125+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:00.867125+0000', 'last_peered': '2024-09-10T20:26:00.867125+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:00.867125+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8361, 'log_dups_size': 0, 'ondisk_log_size': 8361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32208032, 'num_objects': 3932, 'num_object_clones': 0, 'num_object_copies': 15728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8359, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1932}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8230", 'reported_seq': 8503, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.640882+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:01.640882+0000', 'last_peered': '2024-09-10T20:26:01.640882+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:01.640882+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8230, 'log_dups_size': 0, 'ondisk_log_size': 8230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31805168, 'num_objects': 3883, 'num_object_clones': 0, 'num_object_copies': 15532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8227, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2027}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8278", 'reported_seq': 8574, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.802548+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:01.802548+0000', 'last_peered': '2024-09-10T20:26:01.802548+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:01.802548+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8278, 'log_dups_size': 0, 'ondisk_log_size': 8278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31840848, 'num_objects': 3887, 'num_object_clones': 0, 'num_object_copies': 15548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8277, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8010", 'reported_seq': 8193, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:58.648121+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:25:58.648121+0000', 'last_peered': '2024-09-10T20:25:58.648121+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:25:58.648121+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8010, 'log_dups_size': 0, 'ondisk_log_size': 8010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31300176, 'num_objects': 3821, 'num_object_clones': 0, 'num_object_copies': 15284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8009, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1918}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:02.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8241", 'reported_seq': 8509, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.709892+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:01.709892+0000', 'last_peered': '2024-09-10T20:26:01.709892+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:01.709892+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8241, 'log_dups_size': 0, 'ondisk_log_size': 8241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31560864, 'num_objects': 3853, 'num_object_clones': 0, 'num_object_copies': 15412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8239, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:02.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:02.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8366", 'reported_seq': 8571, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:25:58.511245+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:25:58.511245+0000', 'last_peered': '2024-09-10T20:25:58.511245+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:25:58.511245+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8366, 'log_dups_size': 0, 'ondisk_log_size': 8366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33445024, 'num_objects': 4083, 'num_object_clones': 0, 'num_object_copies': 16332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8364, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:02.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:02.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8224", 'reported_seq': 10339, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:01.096842+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:01.096842+0000', 'last_peered': '2024-09-10T20:26:01.096842+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:01.096842+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8224, 'log_dups_size': 0, 'ondisk_log_size': 8224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31954080, 'num_objects': 3901, 'num_object_clones': 0, 'num_object_copies': 15604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8222, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1020, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:02.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:03.366 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:03.366 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21749/250640 objects degraded (8.677%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":452,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62660,"data_bytes":513274888,"bytes_used":2868248576,"bytes_avail":383678808064,"bytes_total":386547056640,"degraded_objects":21749,"degraded_total":250640,"degraded_ratio":0.086773858921161823,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":347,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:03.385 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:03.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:03.876 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:03.901 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-10T20:26:02.020306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513274888,"num_objects":62660,"num_object_clones":0,"num_object_copies":250640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111685,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131811,"ondisk_log_size":131811,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801024,"kb_used_data":1007424,"kb_used_omap":14,"kb_used_meta":1793457,"kb_avail":374686336,"statfs":{"total":386547056640,"available":383678808064,"internally_reserved":0,"allocated":1031602176,"data_stored":1030749932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1836500534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,7,10,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25829376,"num_objects":-3153,"num_object_clones":0,"num_object_copies":-12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008599"},"pg_stats":[{"pgid":"1.f","version":"40'8247","reported_seq":8528,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.617485+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:01.617485+0000","last_peered":"2024-09-10T20:26:01.617485+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:01.617485+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8247,"log_dups_size":0,"ondisk_log_size":8247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31744000,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8286","reported_seq":8417,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:59.585750+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:25:59.585750+0000","last_peered":"2024-09-10T20:25:59.585750+0000","last_clean":"2024-09-10T20:25:59.585750+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:25:59.585750+0000","last_undegraded":"2024-09-10T20:25:59.585750+0000","last_fullsized":"2024-09-10T20:25:59.585750+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8286,"log_dups_size":0,"ondisk_log_size":8286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32489472,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8200","reported_seq":8403,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.401912+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:25:58.401912+0000","last_peered":"2024-09-10T20:25:58.401912+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:25:58.401912+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275048,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8199,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8381","reported_seq":12550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:59.419664+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:25:59.419664+0000","last_peered":"2024-09-10T20:25:59.419664+0000","last_clean":"2024-09-10T20:25:59.419664+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:25:59.419664+0000","last_undegraded":"2024-09-10T20:25:59.419664+0000","last_fullsized":"2024-09-10T20:25:59.419664+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32972800,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2641,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8314","reported_seq":8632,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.760951+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:01.760951+0000","last_peered":"2024-09-10T20:26:01.760951+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:01.760951+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8314,"log_dups_size":0,"ondisk_log_size":8314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8194","reported_seq":8415,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.708683+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:01.708683+0000","last_peered":"2024-09-10T20:26:01.708683+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:01.708683+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8194,"log_dups_size":0,"ondisk_log_size":8194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":3904,"num_object_clones":0,"num_object_copies":15616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8195","reported_seq":8398,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.074643+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:01.074643+0000","last_peered":"2024-09-10T20:26:01.074643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:01.074643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31741088,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1982}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8361","reported_seq":8529,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:00.867125+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:00.867125+0000","last_peered":"2024-09-10T20:26:00.867125+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:00.867125+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8142","reported_seq":12113,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:01.797029+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:01.797029+0000","last_peered":"2024-09-10T20:26:01.797029+0000","last_clean":"2024-09-10T20:26:01.797029+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:01.797029+0000","last_undegraded":"2024-09-10T20:26:01.797029+0000","last_fullsized":"2024-09-10T20:26:01.797029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31611472,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8142","reported_seq":12112,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:25:58.603336+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:25:58.603336+0000","last_peered":"2024-09-10T20:25:58.603336+0000","last_clean":"2024-09-10T20:25:58.603336+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:25:58.603336+0000","last_undegraded":"2024-09-10T20:25:58.603336+0000","last_fullsized":"2024-09-10T20:25:58.603336+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8230","reported_seq":8503,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.640882+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:01.640882+0000","last_peered":"2024-09-10T20:26:01.640882+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:01.640882+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31805168,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8278","reported_seq":8574,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.802548+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:01.802548+0000","last_peered":"2024-09-10T20:26:01.802548+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:01.802548+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31840848,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8010","reported_seq":8193,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.648121+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:25:58.648121+0000","last_peered":"2024-09-10T20:25:58.648121+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:25:58.648121+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8010,"log_dups_size":0,"ondisk_log_size":8010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300176,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8241","reported_seq":8509,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.709892+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:01.709892+0000","last_peered":"2024-09-10T20:26:01.709892+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:01.709892+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560864,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8366","reported_seq":8571,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:25:58.511245+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:25:58.511245+0000","last_peered":"2024-09-10T20:25:58.511245+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:25:58.511245+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33445024,"num_objects":4083,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8224","reported_seq":10339,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.096842+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:01.096842+0000","last_peered":"2024-09-10T20:26:01.096842+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:01.096842+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8224,"log_dups_size":0,"ondisk_log_size":8224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31954080,"num_objects":3901,"num_object_clones":0,"num_object_copies":15604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513274888,"num_objects":62660,"num_object_clones":0,"num_object_copies":250640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111685,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1028038656,"data_stored":1028038656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131811,"ondisk_log_size":131811,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723156,"kb_used_data":249988,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93648684,"statfs":{"total":96636764160,"available":95896252416,"internally_reserved":0,"allocated":255987712,"data_stored":255774651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726056,"kb_used_data":253368,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93645784,"statfs":{"total":96636764160,"available":95893282816,"internally_reserved":0,"allocated":259448832,"data_stored":259235771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627680,"kb_used_data":253068,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93744160,"statfs":{"total":96636764160,"available":95994019840,"internally_reserved":0,"allocated":259141632,"data_stored":258928571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":383577810},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724132,"kb_used_data":251000,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93647708,"statfs":{"total":96636764160,"available":95895252992,"internally_reserved":0,"allocated":257024000,"data_stored":256810939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256077824,"data_stored":256077824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257912832,"data_stored":257912832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258674688,"data_stored":258674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":255373312,"data_stored":255373312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:06.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:07.566 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:07.566 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:07.593 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-10T20:26:06.021979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":509555720,"num_objects":62206,"num_object_clones":0,"num_object_copies":248824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112139,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132265,"ondisk_log_size":132265,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2783192,"kb_used_data":997948,"kb_used_omap":14,"kb_used_meta":1785137,"kb_avail":374704168,"statfs":{"total":386547056640,"available":383697068032,"internally_reserved":0,"allocated":1021898752,"data_stored":1021046508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1827980854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,6,18,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21274624,"num_objects":-2597,"num_object_clones":0,"num_object_copies":-10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008471"},"pg_stats":[{"pgid":"1.f","version":"40'8247","reported_seq":8528,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.617485+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:01.617485+0000","last_peered":"2024-09-10T20:26:01.617485+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:01.617485+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8247,"log_dups_size":0,"ondisk_log_size":8247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31744000,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8364","reported_seq":8495,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:04.659576+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:04.659576+0000","last_peered":"2024-09-10T20:26:04.659576+0000","last_clean":"2024-09-10T20:26:04.659576+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:04.659576+0000","last_undegraded":"2024-09-10T20:26:04.659576+0000","last_fullsized":"2024-09-10T20:26:04.659576+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":3888,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8260","reported_seq":8463,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.420739+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:03.420739+0000","last_peered":"2024-09-10T20:26:03.420739+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:03.420739+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8260,"log_dups_size":0,"ondisk_log_size":8260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783528,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8259,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8466","reported_seq":12635,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:04.628270+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:04.628270+0000","last_peered":"2024-09-10T20:26:04.628270+0000","last_clean":"2024-09-10T20:26:04.628270+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:04.628270+0000","last_undegraded":"2024-09-10T20:26:04.628270+0000","last_fullsized":"2024-09-10T20:26:04.628270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8466,"log_dups_size":0,"ondisk_log_size":8466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32276480,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8314","reported_seq":8632,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.760951+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:01.760951+0000","last_peered":"2024-09-10T20:26:01.760951+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:01.760951+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8314,"log_dups_size":0,"ondisk_log_size":8314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8194","reported_seq":8415,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.708683+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:01.708683+0000","last_peered":"2024-09-10T20:26:01.708683+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:01.708683+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8194,"log_dups_size":0,"ondisk_log_size":8194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":3904,"num_object_clones":0,"num_object_copies":15616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8195","reported_seq":8398,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.074643+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:01.074643+0000","last_peered":"2024-09-10T20:26:01.074643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:01.074643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31741088,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1982}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8361","reported_seq":8529,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:00.867125+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:00.867125+0000","last_peered":"2024-09-10T20:26:00.867125+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:00.867125+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8142","reported_seq":12113,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:01.797029+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:01.797029+0000","last_peered":"2024-09-10T20:26:01.797029+0000","last_clean":"2024-09-10T20:26:01.797029+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:01.797029+0000","last_undegraded":"2024-09-10T20:26:01.797029+0000","last_fullsized":"2024-09-10T20:26:01.797029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31611472,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8209","reported_seq":12179,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:03.618352+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:03.618352+0000","last_peered":"2024-09-10T20:26:03.618352+0000","last_clean":"2024-09-10T20:26:03.618352+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:03.618352+0000","last_undegraded":"2024-09-10T20:26:03.618352+0000","last_fullsized":"2024-09-10T20:26:03.618352+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32219136,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":847,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8230","reported_seq":8503,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.640882+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:01.640882+0000","last_peered":"2024-09-10T20:26:01.640882+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:01.640882+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31805168,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8278","reported_seq":8574,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.802548+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:01.802548+0000","last_peered":"2024-09-10T20:26:01.802548+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:01.802548+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31840848,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8087","reported_seq":8270,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.625103+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:03.625103+0000","last_peered":"2024-09-10T20:26:03.625103+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:03.625103+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30669392,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1841}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8241","reported_seq":8509,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.709892+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:01.709892+0000","last_peered":"2024-09-10T20:26:01.709892+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:01.709892+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560864,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8453","reported_seq":8658,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.578859+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:03.578859+0000","last_peered":"2024-09-10T20:26:03.578859+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:03.578859+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32732320,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8224","reported_seq":10339,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:01.096842+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:01.096842+0000","last_peered":"2024-09-10T20:26:01.096842+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:01.096842+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8224,"log_dups_size":0,"ondisk_log_size":8224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31954080,"num_objects":3901,"num_object_clones":0,"num_object_copies":15604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":509555720,"num_objects":62206,"num_object_clones":0,"num_object_copies":248824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112139,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1018302464,"data_stored":1018302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132265,"ondisk_log_size":132265,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723156,"kb_used_data":249988,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93648684,"statfs":{"total":96636764160,"available":95896252416,"internally_reserved":0,"allocated":255987712,"data_stored":255774651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708752,"kb_used_data":248576,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93663088,"statfs":{"total":96636764160,"available":95911002112,"internally_reserved":0,"allocated":254541824,"data_stored":254328763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,9,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627152,"kb_used_data":248384,"kb_used_omap":4,"kb_used_meta":378747,"kb_avail":93744688,"statfs":{"total":96636764160,"available":95994560512,"internally_reserved":0,"allocated":254345216,"data_stored":254132155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":387837650},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724132,"kb_used_data":251000,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93647708,"statfs":{"total":96636764160,"available":95895252992,"internally_reserved":0,"allocated":257024000,"data_stored":256810939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256077824,"data_stored":256077824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253079552,"data_stored":253079552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":253771776,"data_stored":253771776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":255373312,"data_stored":255373312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:07.595 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:08.136 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:08.136 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:08.157 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-10T20:26:08.022802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503370760,"num_objects":61451,"num_object_clones":0,"num_object_copies":245804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112894,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133020,"ondisk_log_size":133020,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748932,"kb_used_data":988640,"kb_used_omap":14,"kb_used_meta":1760177,"kb_avail":374738428,"statfs":{"total":386547056640,"available":383732150272,"internally_reserved":0,"allocated":1012367360,"data_stored":1011515116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1802421814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,10,16,9,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26247168,"num_objects":-3204,"num_object_clones":0,"num_object_copies":-12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008459"},"pg_stats":[{"pgid":"1.f","version":"40'8315","reported_seq":8596,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.609720+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:06.609720+0000","last_peered":"2024-09-10T20:26:06.609720+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:06.609720+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31186944,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8315,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1952}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8364","reported_seq":8495,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:04.659576+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:04.659576+0000","last_peered":"2024-09-10T20:26:04.659576+0000","last_clean":"2024-09-10T20:26:04.659576+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:04.659576+0000","last_undegraded":"2024-09-10T20:26:04.659576+0000","last_fullsized":"2024-09-10T20:26:04.659576+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":3888,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8260","reported_seq":8463,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.420739+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:03.420739+0000","last_peered":"2024-09-10T20:26:03.420739+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:03.420739+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8260,"log_dups_size":0,"ondisk_log_size":8260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783528,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8259,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8466","reported_seq":12635,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:04.628270+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:04.628270+0000","last_peered":"2024-09-10T20:26:04.628270+0000","last_clean":"2024-09-10T20:26:04.628270+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:04.628270+0000","last_undegraded":"2024-09-10T20:26:04.628270+0000","last_fullsized":"2024-09-10T20:26:04.628270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8466,"log_dups_size":0,"ondisk_log_size":8466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32276480,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8376","reported_seq":8696,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.705254+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:06.705254+0000","last_peered":"2024-09-10T20:26:06.705254+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:06.705254+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8376,"log_dups_size":0,"ondisk_log_size":8376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31069344,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8374,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8280","reported_seq":8501,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.684046+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:06.684046+0000","last_peered":"2024-09-10T20:26:06.684046+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:06.684046+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8269","reported_seq":8472,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.163321+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:06.163321+0000","last_peered":"2024-09-10T20:26:06.163321+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:06.163321+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134880,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8267,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1908}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8438","reported_seq":8606,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.223674+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:06.223674+0000","last_peered":"2024-09-10T20:26:06.223674+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:06.223674+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8438,"log_dups_size":0,"ondisk_log_size":8438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8214","reported_seq":12186,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:06.781682+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:06.781682+0000","last_peered":"2024-09-10T20:26:06.781682+0000","last_clean":"2024-09-10T20:26:06.781682+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:06.781682+0000","last_undegraded":"2024-09-10T20:26:06.781682+0000","last_fullsized":"2024-09-10T20:26:06.781682+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8214,"log_dups_size":0,"ondisk_log_size":8214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31021648,"num_objects":3787,"num_object_clones":0,"num_object_copies":15148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4767,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8209","reported_seq":12179,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:03.618352+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:03.618352+0000","last_peered":"2024-09-10T20:26:03.618352+0000","last_clean":"2024-09-10T20:26:03.618352+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:03.618352+0000","last_undegraded":"2024-09-10T20:26:03.618352+0000","last_fullsized":"2024-09-10T20:26:03.618352+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32219136,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":847,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8310","reported_seq":8583,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.827311+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:06.827311+0000","last_peered":"2024-09-10T20:26:06.827311+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:06.827311+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8310,"log_dups_size":0,"ondisk_log_size":8310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31149808,"num_objects":3803,"num_object_clones":0,"num_object_copies":15212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8360","reported_seq":8657,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.849893+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:06.849893+0000","last_peered":"2024-09-10T20:26:06.849893+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:06.849893+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31169104,"num_objects":3805,"num_object_clones":0,"num_object_copies":15220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8087","reported_seq":8270,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.625103+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:03.625103+0000","last_peered":"2024-09-10T20:26:03.625103+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:03.625103+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30669392,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1841}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8313","reported_seq":8579,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.851625+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:06.851625+0000","last_peered":"2024-09-10T20:26:06.851625+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:06.851625+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30971040,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1906}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8453","reported_seq":8658,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.578859+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:03.578859+0000","last_peered":"2024-09-10T20:26:03.578859+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:03.578859+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32732320,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8306","reported_seq":10634,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.232229+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:06.232229+0000","last_peered":"2024-09-10T20:26:06.232229+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:06.232229+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31282336,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1125,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503370760,"num_objects":61451,"num_object_clones":0,"num_object_copies":245804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112894,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1008590848,"data_stored":1008590848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133020,"ondisk_log_size":133020,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117076,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706100,"kb_used_data":245392,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93665740,"statfs":{"total":96636764160,"available":95913717760,"internally_reserved":0,"allocated":251281408,"data_stored":251068347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708752,"kb_used_data":248576,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93663088,"statfs":{"total":96636764160,"available":95911002112,"internally_reserved":0,"allocated":254541824,"data_stored":254328763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,9,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627152,"kb_used_data":248384,"kb_used_omap":4,"kb_used_meta":378747,"kb_avail":93744688,"statfs":{"total":96636764160,"available":95994560512,"internally_reserved":0,"allocated":254345216,"data_stored":254132155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":387837650},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215189,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706928,"kb_used_data":246288,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93664912,"statfs":{"total":96636764160,"available":95912869888,"internally_reserved":0,"allocated":252198912,"data_stored":251985851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":251191296,"data_stored":251191296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253079552,"data_stored":253079552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":253771776,"data_stored":253771776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250548224,"data_stored":250548224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:08.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:08.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8315", 'reported_seq': 8596, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.609720+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:06.609720+0000', 'last_peered': '2024-09-10T20:26:06.609720+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:06.609720+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8315, 'log_dups_size': 0, 'ondisk_log_size': 8315, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31186944, 'num_objects': 3807, 'num_object_clones': 0, 'num_object_copies': 15228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8315, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1952}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:08.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:08.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8260", 'reported_seq': 8463, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:03.420739+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:03.420739+0000', 'last_peered': '2024-09-10T20:26:03.420739+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:03.420739+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8260, 'log_dups_size': 0, 'ondisk_log_size': 8260, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31783528, 'num_objects': 3881, 'num_object_clones': 0, 'num_object_copies': 15524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3881, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8259, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1973}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:08.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8376", 'reported_seq': 8696, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.705254+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:06.705254+0000', 'last_peered': '2024-09-10T20:26:06.705254+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:06.705254+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8376, 'log_dups_size': 0, 'ondisk_log_size': 8376, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31069344, 'num_objects': 3793, 'num_object_clones': 0, 'num_object_copies': 15172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8374, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1938}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8280", 'reported_seq': 8501, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.684046+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:06.684046+0000', 'last_peered': '2024-09-10T20:26:06.684046+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:06.684046+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8280, 'log_dups_size': 0, 'ondisk_log_size': 8280, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31277056, 'num_objects': 3818, 'num_object_clones': 0, 'num_object_copies': 15272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8280, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8269", 'reported_seq': 8472, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.163321+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:06.163321+0000', 'last_peered': '2024-09-10T20:26:06.163321+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:06.163321+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8269, 'log_dups_size': 0, 'ondisk_log_size': 8269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31134880, 'num_objects': 3801, 'num_object_clones': 0, 'num_object_copies': 15204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8267, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1908}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8438", 'reported_seq': 8606, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.223674+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:06.223674+0000', 'last_peered': '2024-09-10T20:26:06.223674+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:06.223674+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8438, 'log_dups_size': 0, 'ondisk_log_size': 8438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31577248, 'num_objects': 3855, 'num_object_clones': 0, 'num_object_copies': 15420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8436, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8310", 'reported_seq': 8583, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.827311+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:06.827311+0000', 'last_peered': '2024-09-10T20:26:06.827311+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:06.827311+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8310, 'log_dups_size': 0, 'ondisk_log_size': 8310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31149808, 'num_objects': 3803, 'num_object_clones': 0, 'num_object_copies': 15212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8307, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1947}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:08.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8360", 'reported_seq': 8657, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.849893+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:06.849893+0000', 'last_peered': '2024-09-10T20:26:06.849893+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:06.849893+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8360, 'log_dups_size': 0, 'ondisk_log_size': 8360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31169104, 'num_objects': 3805, 'num_object_clones': 0, 'num_object_copies': 15220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8359, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1945}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8087", 'reported_seq': 8270, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:03.625103+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:03.625103+0000', 'last_peered': '2024-09-10T20:26:03.625103+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:03.625103+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8087, 'log_dups_size': 0, 'ondisk_log_size': 8087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30669392, 'num_objects': 3744, 'num_object_clones': 0, 'num_object_copies': 14976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8086, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8313", 'reported_seq': 8579, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.851625+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:06.851625+0000', 'last_peered': '2024-09-10T20:26:06.851625+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:06.851625+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8313, 'log_dups_size': 0, 'ondisk_log_size': 8313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30971040, 'num_objects': 3781, 'num_object_clones': 0, 'num_object_copies': 15124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8311, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1906}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8453", 'reported_seq': 8658, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:03.578859+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:03.578859+0000', 'last_peered': '2024-09-10T20:26:03.578859+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:03.578859+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8453, 'log_dups_size': 0, 'ondisk_log_size': 8453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32732320, 'num_objects': 3996, 'num_object_clones': 0, 'num_object_copies': 15984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8451, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:08.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8306", 'reported_seq': 10634, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:06.232229+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:06.232229+0000', 'last_peered': '2024-09-10T20:26:06.232229+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:06.232229+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8306, 'log_dups_size': 0, 'ondisk_log_size': 8306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31282336, 'num_objects': 3819, 'num_object_clones': 0, 'num_object_copies': 15276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 858, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8304, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1125, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2961}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:08.161 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:08.965 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:08.965 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21644/245804 objects degraded (8.805%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":457,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61451,"data_bytes":503370760,"bytes_used":2814906368,"bytes_avail":383732150272,"bytes_total":386547056640,"degraded_objects":21644,"degraded_total":245804,"degraded_ratio":0.088053896600543524,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":304,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":266},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:08.987 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:08.987 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:09.611 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:09.612 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:09.641 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-10T20:26:08.022802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503370760,"num_objects":61451,"num_object_clones":0,"num_object_copies":245804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112894,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133020,"ondisk_log_size":133020,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748932,"kb_used_data":988640,"kb_used_omap":14,"kb_used_meta":1760177,"kb_avail":374738428,"statfs":{"total":386547056640,"available":383732150272,"internally_reserved":0,"allocated":1012367360,"data_stored":1011515116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1802421814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,10,16,9,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26247168,"num_objects":-3204,"num_object_clones":0,"num_object_copies":-12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008459"},"pg_stats":[{"pgid":"1.f","version":"40'8315","reported_seq":8596,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.609720+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:06.609720+0000","last_peered":"2024-09-10T20:26:06.609720+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:06.609720+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31186944,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8315,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1952}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8364","reported_seq":8495,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:04.659576+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:04.659576+0000","last_peered":"2024-09-10T20:26:04.659576+0000","last_clean":"2024-09-10T20:26:04.659576+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:04.659576+0000","last_undegraded":"2024-09-10T20:26:04.659576+0000","last_fullsized":"2024-09-10T20:26:04.659576+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":3888,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8260","reported_seq":8463,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.420739+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:03.420739+0000","last_peered":"2024-09-10T20:26:03.420739+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:03.420739+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8260,"log_dups_size":0,"ondisk_log_size":8260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783528,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8259,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8466","reported_seq":12635,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:04.628270+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:04.628270+0000","last_peered":"2024-09-10T20:26:04.628270+0000","last_clean":"2024-09-10T20:26:04.628270+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:04.628270+0000","last_undegraded":"2024-09-10T20:26:04.628270+0000","last_fullsized":"2024-09-10T20:26:04.628270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8466,"log_dups_size":0,"ondisk_log_size":8466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32276480,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8376","reported_seq":8696,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.705254+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:06.705254+0000","last_peered":"2024-09-10T20:26:06.705254+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:06.705254+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8376,"log_dups_size":0,"ondisk_log_size":8376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31069344,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8374,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8280","reported_seq":8501,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.684046+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:06.684046+0000","last_peered":"2024-09-10T20:26:06.684046+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:06.684046+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8269","reported_seq":8472,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.163321+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:06.163321+0000","last_peered":"2024-09-10T20:26:06.163321+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:06.163321+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134880,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8267,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1908}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8438","reported_seq":8606,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.223674+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:06.223674+0000","last_peered":"2024-09-10T20:26:06.223674+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:06.223674+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8438,"log_dups_size":0,"ondisk_log_size":8438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8214","reported_seq":12186,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:06.781682+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:06.781682+0000","last_peered":"2024-09-10T20:26:06.781682+0000","last_clean":"2024-09-10T20:26:06.781682+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:06.781682+0000","last_undegraded":"2024-09-10T20:26:06.781682+0000","last_fullsized":"2024-09-10T20:26:06.781682+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8214,"log_dups_size":0,"ondisk_log_size":8214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31021648,"num_objects":3787,"num_object_clones":0,"num_object_copies":15148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4767,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8209","reported_seq":12179,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:03.618352+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:03.618352+0000","last_peered":"2024-09-10T20:26:03.618352+0000","last_clean":"2024-09-10T20:26:03.618352+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:03.618352+0000","last_undegraded":"2024-09-10T20:26:03.618352+0000","last_fullsized":"2024-09-10T20:26:03.618352+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32219136,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":847,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8310","reported_seq":8583,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.827311+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:06.827311+0000","last_peered":"2024-09-10T20:26:06.827311+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:06.827311+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8310,"log_dups_size":0,"ondisk_log_size":8310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31149808,"num_objects":3803,"num_object_clones":0,"num_object_copies":15212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8360","reported_seq":8657,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.849893+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:06.849893+0000","last_peered":"2024-09-10T20:26:06.849893+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:06.849893+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31169104,"num_objects":3805,"num_object_clones":0,"num_object_copies":15220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8087","reported_seq":8270,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.625103+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:03.625103+0000","last_peered":"2024-09-10T20:26:03.625103+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:03.625103+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30669392,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1841}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8313","reported_seq":8579,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.851625+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:06.851625+0000","last_peered":"2024-09-10T20:26:06.851625+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:06.851625+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30971040,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1906}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8453","reported_seq":8658,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:03.578859+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:03.578859+0000","last_peered":"2024-09-10T20:26:03.578859+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:03.578859+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32732320,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8306","reported_seq":10634,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:06.232229+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:06.232229+0000","last_peered":"2024-09-10T20:26:06.232229+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:06.232229+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31282336,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1125,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503370760,"num_objects":61451,"num_object_clones":0,"num_object_copies":245804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112894,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1008590848,"data_stored":1008590848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133020,"ondisk_log_size":133020,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117076,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706100,"kb_used_data":245392,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93665740,"statfs":{"total":96636764160,"available":95913717760,"internally_reserved":0,"allocated":251281408,"data_stored":251068347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708752,"kb_used_data":248576,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93663088,"statfs":{"total":96636764160,"available":95911002112,"internally_reserved":0,"allocated":254541824,"data_stored":254328763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,9,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627152,"kb_used_data":248384,"kb_used_omap":4,"kb_used_meta":378747,"kb_avail":93744688,"statfs":{"total":96636764160,"available":95994560512,"internally_reserved":0,"allocated":254345216,"data_stored":254132155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":387837650},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215189,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706928,"kb_used_data":246288,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93664912,"statfs":{"total":96636764160,"available":95912869888,"internally_reserved":0,"allocated":252198912,"data_stored":251985851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":251191296,"data_stored":251191296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253079552,"data_stored":253079552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":253771776,"data_stored":253771776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250548224,"data_stored":250548224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:12.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:13.278 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:13.278 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:13.306 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-10T20:26:12.024169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493736968,"num_objects":60275,"num_object_clones":0,"num_object_copies":241100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114070,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134196,"ondisk_log_size":134196,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745768,"kb_used_data":968788,"kb_used_omap":14,"kb_used_meta":1776817,"kb_avail":374741592,"statfs":{"total":386547056640,"available":383735390208,"internally_reserved":0,"allocated":992038912,"data_stored":991186668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1819461174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,0,1,15,9,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25944064,"num_objects":-3167,"num_object_clones":0,"num_object_copies":-12668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3167,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"1.f","version":"40'8382","reported_seq":8663,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.876190+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:11.876190+0000","last_peered":"2024-09-10T20:26:11.876190+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:11.876190+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8382,"log_dups_size":0,"ondisk_log_size":8382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30638080,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8382,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8444","reported_seq":8576,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:09.784990+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:09.784990+0000","last_peered":"2024-09-10T20:26:09.784990+0000","last_clean":"2024-09-10T20:26:09.784990+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:09.784990+0000","last_undegraded":"2024-09-10T20:26:09.784990+0000","last_fullsized":"2024-09-10T20:26:09.784990+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8336","reported_seq":8539,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:08.644864+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:08.644864+0000","last_peered":"2024-09-10T20:26:08.644864+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:08.644864+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31160936,"num_objects":3805,"num_object_clones":0,"num_object_copies":15220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8335,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8543","reported_seq":12712,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:09.717028+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:09.717028+0000","last_peered":"2024-09-10T20:26:09.717028+0000","last_clean":"2024-09-10T20:26:09.717028+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:09.717028+0000","last_undegraded":"2024-09-10T20:26:09.717028+0000","last_fullsized":"2024-09-10T20:26:09.717028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8543,"log_dups_size":0,"ondisk_log_size":8543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31645696,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8451","reported_seq":8771,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.895834+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:11.895834+0000","last_peered":"2024-09-10T20:26:11.895834+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:11.895834+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8451,"log_dups_size":0,"ondisk_log_size":8451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30454944,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8350","reported_seq":8571,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.715832+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:11.715832+0000","last_peered":"2024-09-10T20:26:11.715832+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:11.715832+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30703616,"num_objects":3748,"num_object_clones":0,"num_object_copies":14992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8350,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8344","reported_seq":8547,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.061444+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:11.061444+0000","last_peered":"2024-09-10T20:26:11.061444+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:11.061444+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30520480,"num_objects":3726,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8510","reported_seq":8678,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.300653+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:11.300653+0000","last_peered":"2024-09-10T20:26:11.300653+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:11.300653+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8510,"log_dups_size":0,"ondisk_log_size":8510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987424,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8508,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8268","reported_seq":12240,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:11.838913+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:11.838913+0000","last_peered":"2024-09-10T20:26:11.838913+0000","last_clean":"2024-09-10T20:26:11.838913+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:11.838913+0000","last_undegraded":"2024-09-10T20:26:11.838913+0000","last_fullsized":"2024-09-10T20:26:11.838913+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8268,"log_dups_size":0,"ondisk_log_size":8268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579280,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8284","reported_seq":12254,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:08.597328+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:08.597328+0000","last_peered":"2024-09-10T20:26:08.597328+0000","last_clean":"2024-09-10T20:26:08.597328+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:08.597328+0000","last_undegraded":"2024-09-10T20:26:08.597328+0000","last_fullsized":"2024-09-10T20:26:08.597328+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31604736,"num_objects":3858,"num_object_clones":0,"num_object_copies":15432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8389","reported_seq":8662,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.741221+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:11.741221+0000","last_peered":"2024-09-10T20:26:11.741221+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:11.741221+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30502640,"num_objects":3724,"num_object_clones":0,"num_object_copies":14896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8386,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8445","reported_seq":8742,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.975382+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:11.975382+0000","last_peered":"2024-09-10T20:26:11.975382+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:11.975382+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30472784,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8444,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8160","reported_seq":8342,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:08.763260+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:08.763260+0000","last_peered":"2024-09-10T20:26:08.763260+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:08.763260+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30071376,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8159,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1768}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8386","reported_seq":8655,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.852750+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:11.852750+0000","last_peered":"2024-09-10T20:26:11.852750+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:11.852750+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373024,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8384,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8523","reported_seq":8728,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:08.592902+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:08.592902+0000","last_peered":"2024-09-10T20:26:08.592902+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:08.592902+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8523,"log_dups_size":0,"ondisk_log_size":8523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32158880,"num_objects":3926,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8521,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2018}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8381","reported_seq":10834,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.315403+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:11.315403+0000","last_peered":"2024-09-10T20:26:11.315403+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:11.315403+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30667936,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":493736968,"num_objects":60275,"num_object_clones":0,"num_object_copies":241100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114070,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":988876800,"data_stored":988876800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134196,"ondisk_log_size":134196,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705628,"kb_used_data":240736,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93666212,"statfs":{"total":96636764160,"available":95914201088,"internally_reserved":0,"allocated":246513664,"data_stored":246300603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708204,"kb_used_data":243852,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93663636,"statfs":{"total":96636764160,"available":95911563264,"internally_reserved":0,"allocated":249704448,"data_stored":249491387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625636,"kb_used_data":242704,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93746204,"statfs":{"total":96636764160,"available":95996112896,"internally_reserved":0,"allocated":248528896,"data_stored":248315835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":392097490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,8,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706300,"kb_used_data":241496,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93665540,"statfs":{"total":96636764160,"available":95913512960,"internally_reserved":0,"allocated":247291904,"data_stored":247078843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246292480,"data_stored":246292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248168448,"data_stored":248168448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248807424,"data_stored":248807424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245608448,"data_stored":245608448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:13.307 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:13.887 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:13.888 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:13.917 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-10T20:26:12.024169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493736968,"num_objects":60275,"num_object_clones":0,"num_object_copies":241100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114070,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134196,"ondisk_log_size":134196,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745768,"kb_used_data":968788,"kb_used_omap":14,"kb_used_meta":1776817,"kb_avail":374741592,"statfs":{"total":386547056640,"available":383735390208,"internally_reserved":0,"allocated":992038912,"data_stored":991186668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1819461174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,0,1,15,9,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25944064,"num_objects":-3167,"num_object_clones":0,"num_object_copies":-12668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3167,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"1.f","version":"40'8382","reported_seq":8663,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.876190+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:11.876190+0000","last_peered":"2024-09-10T20:26:11.876190+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:11.876190+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8382,"log_dups_size":0,"ondisk_log_size":8382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30638080,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8382,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8444","reported_seq":8576,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:09.784990+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:09.784990+0000","last_peered":"2024-09-10T20:26:09.784990+0000","last_clean":"2024-09-10T20:26:09.784990+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:09.784990+0000","last_undegraded":"2024-09-10T20:26:09.784990+0000","last_fullsized":"2024-09-10T20:26:09.784990+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8336","reported_seq":8539,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:08.644864+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:08.644864+0000","last_peered":"2024-09-10T20:26:08.644864+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:08.644864+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31160936,"num_objects":3805,"num_object_clones":0,"num_object_copies":15220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8335,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8543","reported_seq":12712,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:09.717028+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:09.717028+0000","last_peered":"2024-09-10T20:26:09.717028+0000","last_clean":"2024-09-10T20:26:09.717028+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:09.717028+0000","last_undegraded":"2024-09-10T20:26:09.717028+0000","last_fullsized":"2024-09-10T20:26:09.717028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8543,"log_dups_size":0,"ondisk_log_size":8543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31645696,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8451","reported_seq":8771,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.895834+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:11.895834+0000","last_peered":"2024-09-10T20:26:11.895834+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:11.895834+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8451,"log_dups_size":0,"ondisk_log_size":8451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30454944,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8350","reported_seq":8571,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.715832+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:11.715832+0000","last_peered":"2024-09-10T20:26:11.715832+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:11.715832+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30703616,"num_objects":3748,"num_object_clones":0,"num_object_copies":14992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8350,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8344","reported_seq":8547,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.061444+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:11.061444+0000","last_peered":"2024-09-10T20:26:11.061444+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:11.061444+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30520480,"num_objects":3726,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8510","reported_seq":8678,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.300653+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:11.300653+0000","last_peered":"2024-09-10T20:26:11.300653+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:11.300653+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8510,"log_dups_size":0,"ondisk_log_size":8510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987424,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8508,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8268","reported_seq":12240,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:11.838913+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:11.838913+0000","last_peered":"2024-09-10T20:26:11.838913+0000","last_clean":"2024-09-10T20:26:11.838913+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:11.838913+0000","last_undegraded":"2024-09-10T20:26:11.838913+0000","last_fullsized":"2024-09-10T20:26:11.838913+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8268,"log_dups_size":0,"ondisk_log_size":8268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579280,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8284","reported_seq":12254,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:08.597328+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:08.597328+0000","last_peered":"2024-09-10T20:26:08.597328+0000","last_clean":"2024-09-10T20:26:08.597328+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:08.597328+0000","last_undegraded":"2024-09-10T20:26:08.597328+0000","last_fullsized":"2024-09-10T20:26:08.597328+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31604736,"num_objects":3858,"num_object_clones":0,"num_object_copies":15432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8389","reported_seq":8662,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.741221+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:11.741221+0000","last_peered":"2024-09-10T20:26:11.741221+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:11.741221+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30502640,"num_objects":3724,"num_object_clones":0,"num_object_copies":14896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8386,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8445","reported_seq":8742,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.975382+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:11.975382+0000","last_peered":"2024-09-10T20:26:11.975382+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:11.975382+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30472784,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8444,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8160","reported_seq":8342,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:08.763260+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:08.763260+0000","last_peered":"2024-09-10T20:26:08.763260+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:08.763260+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30071376,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8159,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1768}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8386","reported_seq":8655,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.852750+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:11.852750+0000","last_peered":"2024-09-10T20:26:11.852750+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:11.852750+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373024,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8384,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8523","reported_seq":8728,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:08.592902+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:08.592902+0000","last_peered":"2024-09-10T20:26:08.592902+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:08.592902+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8523,"log_dups_size":0,"ondisk_log_size":8523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32158880,"num_objects":3926,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8521,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2018}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8381","reported_seq":10834,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.315403+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:11.315403+0000","last_peered":"2024-09-10T20:26:11.315403+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:11.315403+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30667936,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":493736968,"num_objects":60275,"num_object_clones":0,"num_object_copies":241100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114070,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":988876800,"data_stored":988876800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134196,"ondisk_log_size":134196,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705628,"kb_used_data":240736,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93666212,"statfs":{"total":96636764160,"available":95914201088,"internally_reserved":0,"allocated":246513664,"data_stored":246300603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708204,"kb_used_data":243852,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93663636,"statfs":{"total":96636764160,"available":95911563264,"internally_reserved":0,"allocated":249704448,"data_stored":249491387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625636,"kb_used_data":242704,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93746204,"statfs":{"total":96636764160,"available":95996112896,"internally_reserved":0,"allocated":248528896,"data_stored":248315835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":392097490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,8,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706300,"kb_used_data":241496,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93665540,"statfs":{"total":96636764160,"available":95913512960,"internally_reserved":0,"allocated":247291904,"data_stored":247078843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246292480,"data_stored":246292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248168448,"data_stored":248168448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248807424,"data_stored":248807424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245608448,"data_stored":245608448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:13.919 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:13.919 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8382", 'reported_seq': 8663, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.876190+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:11.876190+0000', 'last_peered': '2024-09-10T20:26:11.876190+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:11.876190+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8382, 'log_dups_size': 0, 'ondisk_log_size': 8382, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30638080, 'num_objects': 3740, 'num_object_clones': 0, 'num_object_copies': 14960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8382, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1885}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:13.919 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:13.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8336", 'reported_seq': 8539, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:08.644864+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:08.644864+0000', 'last_peered': '2024-09-10T20:26:08.644864+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:08.644864+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8336, 'log_dups_size': 0, 'ondisk_log_size': 8336, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31160936, 'num_objects': 3805, 'num_object_clones': 0, 'num_object_copies': 15220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3805, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8335, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1897}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:13.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:13.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8451", 'reported_seq': 8771, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.895834+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:11.895834+0000', 'last_peered': '2024-09-10T20:26:11.895834+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:11.895834+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8451, 'log_dups_size': 0, 'ondisk_log_size': 8451, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30454944, 'num_objects': 3718, 'num_object_clones': 0, 'num_object_copies': 14872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8449, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1863}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:13.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:13.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8350", 'reported_seq': 8571, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.715832+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:11.715832+0000', 'last_peered': '2024-09-10T20:26:11.715832+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:11.715832+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8350, 'log_dups_size': 0, 'ondisk_log_size': 8350, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30703616, 'num_objects': 3748, 'num_object_clones': 0, 'num_object_copies': 14992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8350, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1891}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:13.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:13.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8344", 'reported_seq': 8547, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.061444+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:11.061444+0000', 'last_peered': '2024-09-10T20:26:11.061444+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:11.061444+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8344, 'log_dups_size': 0, 'ondisk_log_size': 8344, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30520480, 'num_objects': 3726, 'num_object_clones': 0, 'num_object_copies': 14904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8342, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1833}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:13.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:13.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8510", 'reported_seq': 8678, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.300653+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:11.300653+0000', 'last_peered': '2024-09-10T20:26:11.300653+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:11.300653+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8510, 'log_dups_size': 0, 'ondisk_log_size': 8510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30987424, 'num_objects': 3783, 'num_object_clones': 0, 'num_object_copies': 15132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8508, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:13.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:13.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8389", 'reported_seq': 8662, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.741221+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:11.741221+0000', 'last_peered': '2024-09-10T20:26:11.741221+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:11.741221+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8389, 'log_dups_size': 0, 'ondisk_log_size': 8389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30502640, 'num_objects': 3724, 'num_object_clones': 0, 'num_object_copies': 14896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8386, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1868}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:13.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:13.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8445", 'reported_seq': 8742, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.975382+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:11.975382+0000', 'last_peered': '2024-09-10T20:26:11.975382+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:11.975382+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8445, 'log_dups_size': 0, 'ondisk_log_size': 8445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30472784, 'num_objects': 3720, 'num_object_clones': 0, 'num_object_copies': 14880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8444, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:13.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:13.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8160", 'reported_seq': 8342, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:08.763260+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:08.763260+0000', 'last_peered': '2024-09-10T20:26:08.763260+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:08.763260+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8160, 'log_dups_size': 0, 'ondisk_log_size': 8160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30071376, 'num_objects': 3671, 'num_object_clones': 0, 'num_object_copies': 14684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8159, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1768}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:13.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:13.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8386", 'reported_seq': 8655, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.852750+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:11.852750+0000', 'last_peered': '2024-09-10T20:26:11.852750+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:11.852750+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8386, 'log_dups_size': 0, 'ondisk_log_size': 8386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30373024, 'num_objects': 3708, 'num_object_clones': 0, 'num_object_copies': 14832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3708, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8384, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:13.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:13.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8523", 'reported_seq': 8728, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:08.592902+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:08.592902+0000', 'last_peered': '2024-09-10T20:26:08.592902+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:08.592902+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8523, 'log_dups_size': 0, 'ondisk_log_size': 8523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32158880, 'num_objects': 3926, 'num_object_clones': 0, 'num_object_copies': 15704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8521, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:13.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:13.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8381", 'reported_seq': 10834, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:11.315403+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:11.315403+0000', 'last_peered': '2024-09-10T20:26:11.315403+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:11.315403+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8381, 'log_dups_size': 0, 'ondisk_log_size': 8381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30667936, 'num_objects': 3744, 'num_object_clones': 0, 'num_object_copies': 14976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 796, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8379, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1187, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:13.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:14.653 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21582/239872 objects degraded (8.997%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":463,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59968,"data_bytes":491222024,"bytes_used":2811002880,"bytes_avail":383736053760,"bytes_total":386547056640,"degraded_objects":21582,"degraded_total":239872,"degraded_ratio":0.089972985592315896,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":224},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:14.674 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:15.295 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:15.296 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:15.316 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-10T20:26:14.024849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":491222024,"num_objects":59968,"num_object_clones":0,"num_object_copies":239872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114377,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134503,"ondisk_log_size":134503,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745120,"kb_used_data":963984,"kb_used_omap":14,"kb_used_meta":1780977,"kb_avail":374742240,"statfs":{"total":386547056640,"available":383736053760,"internally_reserved":0,"allocated":987119616,"data_stored":986267372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1823721014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,0,1,16,8,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22052864,"num_objects":-2692,"num_object_clones":0,"num_object_copies":-10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2692,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004542"},"pg_stats":[{"pgid":"1.f","version":"40'8382","reported_seq":8663,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.876190+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:11.876190+0000","last_peered":"2024-09-10T20:26:11.876190+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:11.876190+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8382,"log_dups_size":0,"ondisk_log_size":8382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30638080,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8382,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8444","reported_seq":8576,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:09.784990+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:09.784990+0000","last_peered":"2024-09-10T20:26:09.784990+0000","last_clean":"2024-09-10T20:26:09.784990+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:09.784990+0000","last_undegraded":"2024-09-10T20:26:09.784990+0000","last_fullsized":"2024-09-10T20:26:09.784990+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8406","reported_seq":8607,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.783986+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:13.783986+0000","last_peered":"2024-09-10T20:26:13.783986+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:13.783986+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30587496,"num_objects":3735,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8405,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1827}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8543","reported_seq":12712,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:09.717028+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:09.717028+0000","last_peered":"2024-09-10T20:26:09.717028+0000","last_clean":"2024-09-10T20:26:09.717028+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:09.717028+0000","last_undegraded":"2024-09-10T20:26:09.717028+0000","last_fullsized":"2024-09-10T20:26:09.717028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8543,"log_dups_size":0,"ondisk_log_size":8543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31645696,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8451","reported_seq":8771,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.895834+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:11.895834+0000","last_peered":"2024-09-10T20:26:11.895834+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:11.895834+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8451,"log_dups_size":0,"ondisk_log_size":8451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30454944,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8350","reported_seq":8571,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.715832+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:11.715832+0000","last_peered":"2024-09-10T20:26:11.715832+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:11.715832+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30703616,"num_objects":3748,"num_object_clones":0,"num_object_copies":14992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8350,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8344","reported_seq":8547,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.061444+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:11.061444+0000","last_peered":"2024-09-10T20:26:11.061444+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:11.061444+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30520480,"num_objects":3726,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8510","reported_seq":8678,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.300653+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:11.300653+0000","last_peered":"2024-09-10T20:26:11.300653+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:11.300653+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8510,"log_dups_size":0,"ondisk_log_size":8510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987424,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8508,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8268","reported_seq":12240,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:11.838913+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:11.838913+0000","last_peered":"2024-09-10T20:26:11.838913+0000","last_clean":"2024-09-10T20:26:11.838913+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:11.838913+0000","last_undegraded":"2024-09-10T20:26:11.838913+0000","last_fullsized":"2024-09-10T20:26:11.838913+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8268,"log_dups_size":0,"ondisk_log_size":8268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579280,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8357","reported_seq":12327,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:13.785175+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:13.785175+0000","last_peered":"2024-09-10T20:26:13.785175+0000","last_clean":"2024-09-10T20:26:13.785175+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:13.785175+0000","last_undegraded":"2024-09-10T20:26:13.785175+0000","last_fullsized":"2024-09-10T20:26:13.785175+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31006720,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":995,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8389","reported_seq":8662,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.741221+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:11.741221+0000","last_peered":"2024-09-10T20:26:11.741221+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:11.741221+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30502640,"num_objects":3724,"num_object_clones":0,"num_object_copies":14896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8386,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8445","reported_seq":8742,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.975382+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:11.975382+0000","last_peered":"2024-09-10T20:26:11.975382+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:11.975382+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30472784,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8444,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8242","reported_seq":8425,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.644486+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:13.644486+0000","last_peered":"2024-09-10T20:26:13.644486+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:13.644486+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29399632,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8386","reported_seq":8655,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.852750+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:11.852750+0000","last_peered":"2024-09-10T20:26:11.852750+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:11.852750+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373024,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8384,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8605","reported_seq":8810,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.632106+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:13.632106+0000","last_peered":"2024-09-10T20:26:13.632106+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:13.632106+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487136,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8381","reported_seq":10834,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:11.315403+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:11.315403+0000","last_peered":"2024-09-10T20:26:11.315403+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:11.315403+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30667936,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":491222024,"num_objects":59968,"num_object_clones":0,"num_object_copies":239872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114377,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983871488,"data_stored":983871488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134503,"ondisk_log_size":134503,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705628,"kb_used_data":240736,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93666212,"statfs":{"total":96636764160,"available":95914201088,"internally_reserved":0,"allocated":246513664,"data_stored":246300603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707556,"kb_used_data":239048,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93664284,"statfs":{"total":96636764160,"available":95912226816,"internally_reserved":0,"allocated":244785152,"data_stored":244572091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625636,"kb_used_data":242704,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93746204,"statfs":{"total":96636764160,"available":95996112896,"internally_reserved":0,"allocated":248528896,"data_stored":248315835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":392097490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,8,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706300,"kb_used_data":241496,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93665540,"statfs":{"total":96636764160,"available":95913512960,"internally_reserved":0,"allocated":247291904,"data_stored":247078843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246292480,"data_stored":246292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248168448,"data_stored":248168448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243802112,"data_stored":243802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245608448,"data_stored":245608448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:18.318 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:18.957 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:18.957 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:18.977 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-10T20:26:18.026779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484234248,"num_objects":59115,"num_object_clones":0,"num_object_copies":236460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115230,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2203,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135356,"ondisk_log_size":135356,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749240,"kb_used_data":950080,"kb_used_omap":14,"kb_used_meta":1799025,"kb_avail":374738120,"statfs":{"total":386547056640,"available":383731834880,"internally_reserved":0,"allocated":972881920,"data_stored":972029676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1842202166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,3,14,10,15,3,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25321472,"num_objects":-3091,"num_object_clones":0,"num_object_copies":-12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3091,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004800"},"pg_stats":[{"pgid":"1.f","version":"40'8442","reported_seq":8723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.953575+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:16.953575+0000","last_peered":"2024-09-10T20:26:16.953575+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:16.953575+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":3680,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8442,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8544","reported_seq":8676,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:14.659258+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:14.659258+0000","last_peered":"2024-09-10T20:26:14.659258+0000","last_clean":"2024-09-10T20:26:14.659258+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:14.659258+0000","last_undegraded":"2024-09-10T20:26:14.659258+0000","last_fullsized":"2024-09-10T20:26:14.659258+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30375936,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4985,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8406","reported_seq":8607,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.783986+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:13.783986+0000","last_peered":"2024-09-10T20:26:13.783986+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:13.783986+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30587496,"num_objects":3735,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8405,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1827}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8617","reported_seq":12786,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:14.730377+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:14.730377+0000","last_peered":"2024-09-10T20:26:14.730377+0000","last_clean":"2024-09-10T20:26:14.730377+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:14.730377+0000","last_undegraded":"2024-09-10T20:26:14.730377+0000","last_fullsized":"2024-09-10T20:26:14.730377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8617,"log_dups_size":0,"ondisk_log_size":8617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2877,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8513","reported_seq":8833,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.804703+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:16.804703+0000","last_peered":"2024-09-10T20:26:16.804703+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:16.804703+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29947040,"num_objects":3656,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8430","reported_seq":8652,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:17.001488+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:17.001488+0000","last_peered":"2024-09-10T20:26:17.001488+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:17.001488+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30048256,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8430,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8425","reported_seq":8628,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.270650+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:16.270650+0000","last_peered":"2024-09-10T20:26:16.270650+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:16.270650+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29856928,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8561","reported_seq":8729,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.295383+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:16.295383+0000","last_peered":"2024-09-10T20:26:16.295383+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:16.295383+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8561,"log_dups_size":0,"ondisk_log_size":8561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30569632,"num_objects":3732,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8559,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8344","reported_seq":12316,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:16.963501+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:16.963501+0000","last_peered":"2024-09-10T20:26:16.963501+0000","last_clean":"2024-09-10T20:26:16.963501+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:16.963501+0000","last_undegraded":"2024-09-10T20:26:16.963501+0000","last_fullsized":"2024-09-10T20:26:16.963501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29956688,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8357","reported_seq":12327,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:13.785175+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:13.785175+0000","last_peered":"2024-09-10T20:26:13.785175+0000","last_clean":"2024-09-10T20:26:13.785175+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:13.785175+0000","last_undegraded":"2024-09-10T20:26:13.785175+0000","last_fullsized":"2024-09-10T20:26:13.785175+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31006720,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":995,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8446","reported_seq":8719,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.816668+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:16.816668+0000","last_peered":"2024-09-10T20:26:16.816668+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:16.816668+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8446,"log_dups_size":0,"ondisk_log_size":8446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30035696,"num_objects":3667,"num_object_clones":0,"num_object_copies":14668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8514","reported_seq":8812,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.904614+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:16.904614+0000","last_peered":"2024-09-10T20:26:16.904614+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:16.904614+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8514,"log_dups_size":0,"ondisk_log_size":8514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29907536,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8513,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8242","reported_seq":8425,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.644486+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:13.644486+0000","last_peered":"2024-09-10T20:26:13.644486+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:13.644486+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29399632,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8455","reported_seq":8723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.953505+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:16.953505+0000","last_peered":"2024-09-10T20:26:16.953505+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:16.953505+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8455,"log_dups_size":0,"ondisk_log_size":8455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29807776,"num_objects":3639,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1764}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8605","reported_seq":8810,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.632106+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:13.632106+0000","last_peered":"2024-09-10T20:26:13.632106+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:13.632106+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487136,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8455","reported_seq":11126,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.363892+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:16.363892+0000","last_peered":"2024-09-10T20:26:16.363892+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:16.363892+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8455,"log_dups_size":0,"ondisk_log_size":8455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30061728,"num_objects":3670,"num_object_clones":0,"num_object_copies":14680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1296,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":484234248,"num_objects":59115,"num_object_clones":0,"num_object_copies":236460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115230,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2203,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":969633792,"data_stored":969633792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135356,"ondisk_log_size":135356,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705172,"kb_used_data":236152,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93666668,"statfs":{"total":96636764160,"available":95914668032,"internally_reserved":0,"allocated":241819648,"data_stored":241606587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,9,0,0,0,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707556,"kb_used_data":239048,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93664284,"statfs":{"total":96636764160,"available":95912226816,"internally_reserved":0,"allocated":244785152,"data_stored":244572091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630596,"kb_used_data":237908,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93741244,"statfs":{"total":96636764160,"available":95991033856,"internally_reserved":0,"allocated":243617792,"data_stored":243404731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215191,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705916,"kb_used_data":236972,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93665924,"statfs":{"total":96636764160,"available":95913906176,"internally_reserved":0,"allocated":242659328,"data_stored":242446267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,0,2,6,7,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241598464,"data_stored":241598464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243261440,"data_stored":243261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243802112,"data_stored":243802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240971776,"data_stored":240971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:18.978 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:19.597 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:19.628 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-10T20:26:18.026779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484234248,"num_objects":59115,"num_object_clones":0,"num_object_copies":236460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115230,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2203,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135356,"ondisk_log_size":135356,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749240,"kb_used_data":950080,"kb_used_omap":14,"kb_used_meta":1799025,"kb_avail":374738120,"statfs":{"total":386547056640,"available":383731834880,"internally_reserved":0,"allocated":972881920,"data_stored":972029676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1842202166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,3,14,10,15,3,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25321472,"num_objects":-3091,"num_object_clones":0,"num_object_copies":-12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3091,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004800"},"pg_stats":[{"pgid":"1.f","version":"40'8442","reported_seq":8723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.953575+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:16.953575+0000","last_peered":"2024-09-10T20:26:16.953575+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:16.953575+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":3680,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8442,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8544","reported_seq":8676,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:14.659258+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:14.659258+0000","last_peered":"2024-09-10T20:26:14.659258+0000","last_clean":"2024-09-10T20:26:14.659258+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:14.659258+0000","last_undegraded":"2024-09-10T20:26:14.659258+0000","last_fullsized":"2024-09-10T20:26:14.659258+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30375936,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4985,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8406","reported_seq":8607,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.783986+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:13.783986+0000","last_peered":"2024-09-10T20:26:13.783986+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:13.783986+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30587496,"num_objects":3735,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8405,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1827}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8617","reported_seq":12786,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:14.730377+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:14.730377+0000","last_peered":"2024-09-10T20:26:14.730377+0000","last_clean":"2024-09-10T20:26:14.730377+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:14.730377+0000","last_undegraded":"2024-09-10T20:26:14.730377+0000","last_fullsized":"2024-09-10T20:26:14.730377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8617,"log_dups_size":0,"ondisk_log_size":8617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2877,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8513","reported_seq":8833,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.804703+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:16.804703+0000","last_peered":"2024-09-10T20:26:16.804703+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:16.804703+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29947040,"num_objects":3656,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8430","reported_seq":8652,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:17.001488+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:17.001488+0000","last_peered":"2024-09-10T20:26:17.001488+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:17.001488+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30048256,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8430,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8425","reported_seq":8628,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.270650+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:16.270650+0000","last_peered":"2024-09-10T20:26:16.270650+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:16.270650+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29856928,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8561","reported_seq":8729,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.295383+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:16.295383+0000","last_peered":"2024-09-10T20:26:16.295383+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:16.295383+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8561,"log_dups_size":0,"ondisk_log_size":8561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30569632,"num_objects":3732,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8559,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8344","reported_seq":12316,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:16.963501+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:16.963501+0000","last_peered":"2024-09-10T20:26:16.963501+0000","last_clean":"2024-09-10T20:26:16.963501+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:16.963501+0000","last_undegraded":"2024-09-10T20:26:16.963501+0000","last_fullsized":"2024-09-10T20:26:16.963501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29956688,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8357","reported_seq":12327,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:13.785175+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:13.785175+0000","last_peered":"2024-09-10T20:26:13.785175+0000","last_clean":"2024-09-10T20:26:13.785175+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:13.785175+0000","last_undegraded":"2024-09-10T20:26:13.785175+0000","last_fullsized":"2024-09-10T20:26:13.785175+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31006720,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":995,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8446","reported_seq":8719,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.816668+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:16.816668+0000","last_peered":"2024-09-10T20:26:16.816668+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:16.816668+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8446,"log_dups_size":0,"ondisk_log_size":8446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30035696,"num_objects":3667,"num_object_clones":0,"num_object_copies":14668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8514","reported_seq":8812,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.904614+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:16.904614+0000","last_peered":"2024-09-10T20:26:16.904614+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:16.904614+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8514,"log_dups_size":0,"ondisk_log_size":8514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29907536,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8513,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8242","reported_seq":8425,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.644486+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:13.644486+0000","last_peered":"2024-09-10T20:26:13.644486+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:13.644486+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29399632,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8455","reported_seq":8723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.953505+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:16.953505+0000","last_peered":"2024-09-10T20:26:16.953505+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:16.953505+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8455,"log_dups_size":0,"ondisk_log_size":8455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29807776,"num_objects":3639,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1764}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8605","reported_seq":8810,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:13.632106+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:13.632106+0000","last_peered":"2024-09-10T20:26:13.632106+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:13.632106+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487136,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8455","reported_seq":11126,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.363892+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:16.363892+0000","last_peered":"2024-09-10T20:26:16.363892+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:16.363892+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8455,"log_dups_size":0,"ondisk_log_size":8455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30061728,"num_objects":3670,"num_object_clones":0,"num_object_copies":14680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1296,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":484234248,"num_objects":59115,"num_object_clones":0,"num_object_copies":236460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115230,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2203,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":969633792,"data_stored":969633792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135356,"ondisk_log_size":135356,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705172,"kb_used_data":236152,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93666668,"statfs":{"total":96636764160,"available":95914668032,"internally_reserved":0,"allocated":241819648,"data_stored":241606587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,9,0,0,0,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707556,"kb_used_data":239048,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93664284,"statfs":{"total":96636764160,"available":95912226816,"internally_reserved":0,"allocated":244785152,"data_stored":244572091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630596,"kb_used_data":237908,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93741244,"statfs":{"total":96636764160,"available":95991033856,"internally_reserved":0,"allocated":243617792,"data_stored":243404731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215191,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705916,"kb_used_data":236972,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93665924,"statfs":{"total":96636764160,"available":95913906176,"internally_reserved":0,"allocated":242659328,"data_stored":242446267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,0,2,6,7,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241598464,"data_stored":241598464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243261440,"data_stored":243261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243802112,"data_stored":243802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240971776,"data_stored":240971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:19.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:19.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8442", 'reported_seq': 8723, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.953575+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:16.953575+0000', 'last_peered': '2024-09-10T20:26:16.953575+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:16.953575+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8442, 'log_dups_size': 0, 'ondisk_log_size': 8442, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30146560, 'num_objects': 3680, 'num_object_clones': 0, 'num_object_copies': 14720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8442, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1825}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:19.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:19.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8406", 'reported_seq': 8607, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:13.783986+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:13.783986+0000', 'last_peered': '2024-09-10T20:26:13.783986+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:13.783986+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8406, 'log_dups_size': 0, 'ondisk_log_size': 8406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30587496, 'num_objects': 3735, 'num_object_clones': 0, 'num_object_copies': 14940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3735, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8405, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1827}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:19.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:19.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8513", 'reported_seq': 8833, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.804703+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:16.804703+0000', 'last_peered': '2024-09-10T20:26:16.804703+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:16.804703+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8513, 'log_dups_size': 0, 'ondisk_log_size': 8513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29947040, 'num_objects': 3656, 'num_object_clones': 0, 'num_object_copies': 14624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8511, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1801}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8430", 'reported_seq': 8652, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:17.001488+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:17.001488+0000', 'last_peered': '2024-09-10T20:26:17.001488+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:17.001488+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8430, 'log_dups_size': 0, 'ondisk_log_size': 8430, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30048256, 'num_objects': 3668, 'num_object_clones': 0, 'num_object_copies': 14672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8430, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8425", 'reported_seq': 8628, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.270650+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:16.270650+0000', 'last_peered': '2024-09-10T20:26:16.270650+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:16.270650+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8425, 'log_dups_size': 0, 'ondisk_log_size': 8425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29856928, 'num_objects': 3645, 'num_object_clones': 0, 'num_object_copies': 14580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8423, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1752}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8561", 'reported_seq': 8729, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.295383+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:16.295383+0000', 'last_peered': '2024-09-10T20:26:16.295383+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:16.295383+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8561, 'log_dups_size': 0, 'ondisk_log_size': 8561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30569632, 'num_objects': 3732, 'num_object_clones': 0, 'num_object_copies': 14928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8559, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8446", 'reported_seq': 8719, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.816668+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:16.816668+0000', 'last_peered': '2024-09-10T20:26:16.816668+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:16.816668+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8446, 'log_dups_size': 0, 'ondisk_log_size': 8446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30035696, 'num_objects': 3667, 'num_object_clones': 0, 'num_object_copies': 14668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8443, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8514", 'reported_seq': 8812, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.904614+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:16.904614+0000', 'last_peered': '2024-09-10T20:26:16.904614+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:16.904614+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8514, 'log_dups_size': 0, 'ondisk_log_size': 8514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29907536, 'num_objects': 3651, 'num_object_clones': 0, 'num_object_copies': 14604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8513, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:19.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:19.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8242", 'reported_seq': 8425, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:13.644486+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:13.644486+0000', 'last_peered': '2024-09-10T20:26:13.644486+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:13.644486+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8242, 'log_dups_size': 0, 'ondisk_log_size': 8242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29399632, 'num_objects': 3589, 'num_object_clones': 0, 'num_object_copies': 14356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8241, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:19.631 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:19.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8455", 'reported_seq': 8723, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.953505+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:16.953505+0000', 'last_peered': '2024-09-10T20:26:16.953505+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:16.953505+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8455, 'log_dups_size': 0, 'ondisk_log_size': 8455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29807776, 'num_objects': 3639, 'num_object_clones': 0, 'num_object_copies': 14556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8453, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1764}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:19.631 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:19.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8605", 'reported_seq': 8810, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:13.632106+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:13.632106+0000', 'last_peered': '2024-09-10T20:26:13.632106+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:13.632106+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8605, 'log_dups_size': 0, 'ondisk_log_size': 8605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31487136, 'num_objects': 3844, 'num_object_clones': 0, 'num_object_copies': 15376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8603, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:19.631 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:19.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8455", 'reported_seq': 11126, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:16.363892+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:16.363892+0000', 'last_peered': '2024-09-10T20:26:16.363892+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:16.363892+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8455, 'log_dups_size': 0, 'ondisk_log_size': 8455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30061728, 'num_objects': 3670, 'num_object_clones': 0, 'num_object_copies': 14680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 687, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8453, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1296, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2983}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:19.631 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:20.415 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:20.415 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21473/234576 objects degraded (9.154%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":469,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58644,"data_bytes":480375816,"bytes_used":2814222336,"bytes_avail":383732834304,"bytes_total":386547056640,"degraded_objects":21473,"degraded_total":234576,"degraded_ratio":0.091539628947547913,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":233},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:20.446 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:20.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:21.052 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:21.053 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:21.076 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-10T20:26:20.027632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":480375816,"num_objects":58644,"num_object_clones":0,"num_object_copies":234576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115701,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2203,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135827,"ondisk_log_size":135827,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748264,"kb_used_data":940828,"kb_used_omap":14,"kb_used_meta":1807345,"kb_avail":374739096,"statfs":{"total":386547056640,"available":383732834304,"internally_reserved":0,"allocated":963407872,"data_stored":962555628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1850721846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,7,13,22,10,1,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22994944,"num_objects":-2807,"num_object_clones":0,"num_object_copies":-11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2807,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004830"},"pg_stats":[{"pgid":"1.f","version":"40'8442","reported_seq":8723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.953575+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:16.953575+0000","last_peered":"2024-09-10T20:26:16.953575+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:16.953575+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":3680,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8442,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8618","reported_seq":8750,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:19.680016+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:19.680016+0000","last_peered":"2024-09-10T20:26:19.680016+0000","last_clean":"2024-09-10T20:26:19.680016+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:19.680016+0000","last_undegraded":"2024-09-10T20:26:19.680016+0000","last_fullsized":"2024-09-10T20:26:19.680016+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29769728,"num_objects":3634,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8484","reported_seq":8687,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:18.835524+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:18.835524+0000","last_peered":"2024-09-10T20:26:18.835524+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:18.835524+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8484,"log_dups_size":0,"ondisk_log_size":8484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29948520,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8483,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1749}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8687","reported_seq":12856,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:19.825220+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:19.825220+0000","last_peered":"2024-09-10T20:26:19.825220+0000","last_clean":"2024-09-10T20:26:19.825220+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:19.825220+0000","last_undegraded":"2024-09-10T20:26:19.825220+0000","last_fullsized":"2024-09-10T20:26:19.825220+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8687,"log_dups_size":0,"ondisk_log_size":8687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30466048,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8513","reported_seq":8833,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.804703+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:16.804703+0000","last_peered":"2024-09-10T20:26:16.804703+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:16.804703+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29947040,"num_objects":3656,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8430","reported_seq":8652,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:17.001488+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:17.001488+0000","last_peered":"2024-09-10T20:26:17.001488+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:17.001488+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30048256,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8430,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8425","reported_seq":8628,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.270650+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:16.270650+0000","last_peered":"2024-09-10T20:26:16.270650+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:16.270650+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29856928,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8561","reported_seq":8729,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.295383+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:16.295383+0000","last_peered":"2024-09-10T20:26:16.295383+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:16.295383+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8561,"log_dups_size":0,"ondisk_log_size":8561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30569632,"num_objects":3732,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8559,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8344","reported_seq":12316,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:16.963501+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:16.963501+0000","last_peered":"2024-09-10T20:26:16.963501+0000","last_clean":"2024-09-10T20:26:16.963501+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:16.963501+0000","last_undegraded":"2024-09-10T20:26:16.963501+0000","last_fullsized":"2024-09-10T20:26:16.963501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29956688,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8458","reported_seq":12428,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:18.781299+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:18.781299+0000","last_peered":"2024-09-10T20:26:18.781299+0000","last_clean":"2024-09-10T20:26:18.781299+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:18.781299+0000","last_undegraded":"2024-09-10T20:26:18.781299+0000","last_fullsized":"2024-09-10T20:26:18.781299+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30179328,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1096,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8446","reported_seq":8719,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.816668+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:16.816668+0000","last_peered":"2024-09-10T20:26:16.816668+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:16.816668+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8446,"log_dups_size":0,"ondisk_log_size":8446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30035696,"num_objects":3667,"num_object_clones":0,"num_object_copies":14668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8514","reported_seq":8812,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.904614+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:16.904614+0000","last_peered":"2024-09-10T20:26:16.904614+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:16.904614+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8514,"log_dups_size":0,"ondisk_log_size":8514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29907536,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8513,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8315","reported_seq":8497,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:18.856545+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:18.856545+0000","last_peered":"2024-09-10T20:26:18.856545+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:18.856545+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28801616,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8314,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1613}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8455","reported_seq":8723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.953505+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:16.953505+0000","last_peered":"2024-09-10T20:26:16.953505+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:16.953505+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8455,"log_dups_size":0,"ondisk_log_size":8455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29807776,"num_objects":3639,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1764}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8680","reported_seq":8884,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:18.745430+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:18.745430+0000","last_peered":"2024-09-10T20:26:18.745430+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:18.745430+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872736,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8455","reported_seq":11126,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:16.363892+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:16.363892+0000","last_peered":"2024-09-10T20:26:16.363892+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:16.363892+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8455,"log_dups_size":0,"ondisk_log_size":8455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30061728,"num_objects":3670,"num_object_clones":0,"num_object_copies":14680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1296,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":480375816,"num_objects":58644,"num_object_clones":0,"num_object_copies":234576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115701,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2203,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":960077824,"data_stored":960077824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135827,"ondisk_log_size":135827,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705172,"kb_used_data":236152,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93666668,"statfs":{"total":96636764160,"available":95914668032,"internally_reserved":0,"allocated":241819648,"data_stored":241606587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,9,0,0,0,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707060,"kb_used_data":234404,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93664780,"statfs":{"total":96636764160,"available":95912734720,"internally_reserved":0,"allocated":240029696,"data_stored":239816635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630116,"kb_used_data":233300,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93741724,"statfs":{"total":96636764160,"available":95991525376,"internally_reserved":0,"allocated":238899200,"data_stored":238686139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,11,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215191,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705916,"kb_used_data":236972,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93665924,"statfs":{"total":96636764160,"available":95913906176,"internally_reserved":0,"allocated":242659328,"data_stored":242446267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,0,2,6,7,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241598464,"data_stored":241598464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238456832,"data_stored":238456832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":239050752,"data_stored":239050752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240971776,"data_stored":240971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:24.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:24.681 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:24.681 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:24.700 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-10T20:26:24.029339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":471921672,"num_objects":57612,"num_object_clones":0,"num_object_copies":230448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116733,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136859,"ondisk_log_size":136859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729912,"kb_used_data":926616,"kb_used_omap":14,"kb_used_meta":1803185,"kb_avail":374757448,"statfs":{"total":386547056640,"available":383751626752,"internally_reserved":0,"allocated":948854784,"data_stored":948002540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1846462006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,9,12,28,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21815296,"num_objects":-2663,"num_object_clones":0,"num_object_copies":-10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005170"},"pg_stats":[{"pgid":"1.f","version":"40'8519","reported_seq":8801,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.903179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:21.903179+0000","last_peered":"2024-09-10T20:26:21.903179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:21.903179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8519,"log_dups_size":0,"ondisk_log_size":8519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29515776,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8618","reported_seq":8750,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:19.680016+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:19.680016+0000","last_peered":"2024-09-10T20:26:19.680016+0000","last_clean":"2024-09-10T20:26:19.680016+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:19.680016+0000","last_undegraded":"2024-09-10T20:26:19.680016+0000","last_fullsized":"2024-09-10T20:26:19.680016+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29769728,"num_objects":3634,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8552","reported_seq":8755,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.864593+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:23.864593+0000","last_peered":"2024-09-10T20:26:23.864593+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:23.864593+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8552,"log_dups_size":0,"ondisk_log_size":8552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29391464,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8551,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8687","reported_seq":12856,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:19.825220+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:19.825220+0000","last_peered":"2024-09-10T20:26:19.825220+0000","last_clean":"2024-09-10T20:26:19.825220+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:19.825220+0000","last_undegraded":"2024-09-10T20:26:19.825220+0000","last_fullsized":"2024-09-10T20:26:19.825220+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8687,"log_dups_size":0,"ondisk_log_size":8687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30466048,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8597","reported_seq":8917,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:22.035972+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:22.035972+0000","last_peered":"2024-09-10T20:26:22.035972+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:22.035972+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29258912,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8496","reported_seq":8718,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.912069+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:21.912069+0000","last_peered":"2024-09-10T20:26:21.912069+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:21.912069+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29507584,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8496,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8500","reported_seq":8703,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.336420+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:21.336420+0000","last_peered":"2024-09-10T20:26:21.336420+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:21.336420+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1677}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8636","reported_seq":8801,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.353449+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:21.353449+0000","last_peered":"2024-09-10T20:26:21.353449+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:21.353449+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955232,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8427","reported_seq":12397,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:22.013591+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:22.013591+0000","last_peered":"2024-09-10T20:26:22.013591+0000","last_clean":"2024-09-10T20:26:22.013591+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:22.013591+0000","last_undegraded":"2024-09-10T20:26:22.013591+0000","last_fullsized":"2024-09-10T20:26:22.013591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8427,"log_dups_size":0,"ondisk_log_size":8427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29276752,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8526","reported_seq":12494,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:23.907053+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:23.907053+0000","last_peered":"2024-09-10T20:26:23.907053+0000","last_clean":"2024-09-10T20:26:23.907053+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:23.907053+0000","last_undegraded":"2024-09-10T20:26:23.907053+0000","last_fullsized":"2024-09-10T20:26:23.907053+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8508","reported_seq":8781,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.954373+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:21.954373+0000","last_peered":"2024-09-10T20:26:21.954373+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:21.954373+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8508,"log_dups_size":0,"ondisk_log_size":8508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29527792,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8505,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8589","reported_seq":8887,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.955297+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:21.955297+0000","last_peered":"2024-09-10T20:26:21.955297+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:21.955297+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29293136,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8397","reported_seq":8580,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.895748+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:23.895748+0000","last_peered":"2024-09-10T20:26:23.895748+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:23.895748+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28129872,"num_objects":3434,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8520","reported_seq":8789,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.806658+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:21.806658+0000","last_peered":"2024-09-10T20:26:21.806658+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:21.806658+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29275296,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8757","reported_seq":8962,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.986206+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:23.986206+0000","last_peered":"2024-09-10T20:26:23.986206+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:23.986206+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30241952,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8530","reported_seq":11325,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.356351+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:21.356351+0000","last_peered":"2024-09-10T20:26:21.356351+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:21.356351+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447328,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1358,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":471921672,"num_objects":57612,"num_object_clones":0,"num_object_copies":230448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116733,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945348608,"data_stored":945348608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136859,"ondisk_log_size":136859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704620,"kb_used_data":231444,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93667220,"statfs":{"total":96636764160,"available":95915233280,"internally_reserved":0,"allocated":236998656,"data_stored":236785595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689808,"kb_used_data":229608,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93682032,"statfs":{"total":96636764160,"available":95930400768,"internally_reserved":0,"allocated":235118592,"data_stored":234905531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630116,"kb_used_data":233300,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93741724,"statfs":{"total":96636764160,"available":95991525376,"internally_reserved":0,"allocated":238899200,"data_stored":238686139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,11,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705368,"kb_used_data":232264,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93666472,"statfs":{"total":96636764160,"available":95914467328,"internally_reserved":0,"allocated":237838336,"data_stored":237625275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236740608,"data_stored":236740608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238456832,"data_stored":238456832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234061824,"data_stored":234061824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236089344,"data_stored":236089344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:24.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:25.913 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:25.913 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:25.931 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-10T20:26:24.029339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":471921672,"num_objects":57612,"num_object_clones":0,"num_object_copies":230448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116733,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136859,"ondisk_log_size":136859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729912,"kb_used_data":926616,"kb_used_omap":14,"kb_used_meta":1803185,"kb_avail":374757448,"statfs":{"total":386547056640,"available":383751626752,"internally_reserved":0,"allocated":948854784,"data_stored":948002540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1846462006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,9,12,28,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21815296,"num_objects":-2663,"num_object_clones":0,"num_object_copies":-10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005170"},"pg_stats":[{"pgid":"1.f","version":"40'8519","reported_seq":8801,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.903179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:21.903179+0000","last_peered":"2024-09-10T20:26:21.903179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:21.903179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8519,"log_dups_size":0,"ondisk_log_size":8519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29515776,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8618","reported_seq":8750,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:19.680016+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:19.680016+0000","last_peered":"2024-09-10T20:26:19.680016+0000","last_clean":"2024-09-10T20:26:19.680016+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:19.680016+0000","last_undegraded":"2024-09-10T20:26:19.680016+0000","last_fullsized":"2024-09-10T20:26:19.680016+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29769728,"num_objects":3634,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8552","reported_seq":8755,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.864593+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:23.864593+0000","last_peered":"2024-09-10T20:26:23.864593+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:23.864593+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8552,"log_dups_size":0,"ondisk_log_size":8552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29391464,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8551,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8687","reported_seq":12856,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:19.825220+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:19.825220+0000","last_peered":"2024-09-10T20:26:19.825220+0000","last_clean":"2024-09-10T20:26:19.825220+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:19.825220+0000","last_undegraded":"2024-09-10T20:26:19.825220+0000","last_fullsized":"2024-09-10T20:26:19.825220+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8687,"log_dups_size":0,"ondisk_log_size":8687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30466048,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8597","reported_seq":8917,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:22.035972+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:22.035972+0000","last_peered":"2024-09-10T20:26:22.035972+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:22.035972+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29258912,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8496","reported_seq":8718,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.912069+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:21.912069+0000","last_peered":"2024-09-10T20:26:21.912069+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:21.912069+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29507584,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8496,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8500","reported_seq":8703,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.336420+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:21.336420+0000","last_peered":"2024-09-10T20:26:21.336420+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:21.336420+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1677}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8636","reported_seq":8801,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.353449+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:21.353449+0000","last_peered":"2024-09-10T20:26:21.353449+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:21.353449+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955232,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8427","reported_seq":12397,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:22.013591+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:22.013591+0000","last_peered":"2024-09-10T20:26:22.013591+0000","last_clean":"2024-09-10T20:26:22.013591+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:22.013591+0000","last_undegraded":"2024-09-10T20:26:22.013591+0000","last_fullsized":"2024-09-10T20:26:22.013591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8427,"log_dups_size":0,"ondisk_log_size":8427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29276752,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8526","reported_seq":12494,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:23.907053+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:23.907053+0000","last_peered":"2024-09-10T20:26:23.907053+0000","last_clean":"2024-09-10T20:26:23.907053+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:23.907053+0000","last_undegraded":"2024-09-10T20:26:23.907053+0000","last_fullsized":"2024-09-10T20:26:23.907053+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8508","reported_seq":8781,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.954373+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:21.954373+0000","last_peered":"2024-09-10T20:26:21.954373+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:21.954373+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8508,"log_dups_size":0,"ondisk_log_size":8508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29527792,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8505,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8589","reported_seq":8887,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.955297+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:21.955297+0000","last_peered":"2024-09-10T20:26:21.955297+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:21.955297+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29293136,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8397","reported_seq":8580,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.895748+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:23.895748+0000","last_peered":"2024-09-10T20:26:23.895748+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:23.895748+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28129872,"num_objects":3434,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8520","reported_seq":8789,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.806658+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:21.806658+0000","last_peered":"2024-09-10T20:26:21.806658+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:21.806658+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29275296,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8757","reported_seq":8962,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.986206+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:23.986206+0000","last_peered":"2024-09-10T20:26:23.986206+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:23.986206+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30241952,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8530","reported_seq":11325,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.356351+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:21.356351+0000","last_peered":"2024-09-10T20:26:21.356351+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:21.356351+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447328,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1358,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":471921672,"num_objects":57612,"num_object_clones":0,"num_object_copies":230448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116733,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945348608,"data_stored":945348608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136859,"ondisk_log_size":136859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704620,"kb_used_data":231444,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93667220,"statfs":{"total":96636764160,"available":95915233280,"internally_reserved":0,"allocated":236998656,"data_stored":236785595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689808,"kb_used_data":229608,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93682032,"statfs":{"total":96636764160,"available":95930400768,"internally_reserved":0,"allocated":235118592,"data_stored":234905531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630116,"kb_used_data":233300,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93741724,"statfs":{"total":96636764160,"available":95991525376,"internally_reserved":0,"allocated":238899200,"data_stored":238686139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,11,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705368,"kb_used_data":232264,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93666472,"statfs":{"total":96636764160,"available":95914467328,"internally_reserved":0,"allocated":237838336,"data_stored":237625275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236740608,"data_stored":236740608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238456832,"data_stored":238456832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234061824,"data_stored":234061824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236089344,"data_stored":236089344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:25.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:25.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8519", 'reported_seq': 8801, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.903179+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:21.903179+0000', 'last_peered': '2024-09-10T20:26:21.903179+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:21.903179+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8519, 'log_dups_size': 0, 'ondisk_log_size': 8519, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29515776, 'num_objects': 3603, 'num_object_clones': 0, 'num_object_copies': 14412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8519, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1748}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:25.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:25.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8552", 'reported_seq': 8755, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:23.864593+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:23.864593+0000', 'last_peered': '2024-09-10T20:26:23.864593+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:23.864593+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8552, 'log_dups_size': 0, 'ondisk_log_size': 8552, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29391464, 'num_objects': 3589, 'num_object_clones': 0, 'num_object_copies': 14356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3589, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8551, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1681}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8597", 'reported_seq': 8917, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:22.035972+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:22.035972+0000', 'last_peered': '2024-09-10T20:26:22.035972+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:22.035972+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8597, 'log_dups_size': 0, 'ondisk_log_size': 8597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29258912, 'num_objects': 3572, 'num_object_clones': 0, 'num_object_copies': 14288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8595, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1717}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8496", 'reported_seq': 8718, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.912069+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:21.912069+0000', 'last_peered': '2024-09-10T20:26:21.912069+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:21.912069+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8496, 'log_dups_size': 0, 'ondisk_log_size': 8496, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29507584, 'num_objects': 3602, 'num_object_clones': 0, 'num_object_copies': 14408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8496, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8500", 'reported_seq': 8703, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.336420+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:21.336420+0000', 'last_peered': '2024-09-10T20:26:21.336420+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:21.336420+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8500, 'log_dups_size': 0, 'ondisk_log_size': 8500, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29242528, 'num_objects': 3570, 'num_object_clones': 0, 'num_object_copies': 14280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8498, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1677}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8636", 'reported_seq': 8801, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.353449+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:21.353449+0000', 'last_peered': '2024-09-10T20:26:21.353449+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:21.353449+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8636, 'log_dups_size': 0, 'ondisk_log_size': 8636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29955232, 'num_objects': 3657, 'num_object_clones': 0, 'num_object_copies': 14628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8634, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:25.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8508", 'reported_seq': 8781, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.954373+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:21.954373+0000', 'last_peered': '2024-09-10T20:26:21.954373+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:21.954373+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8508, 'log_dups_size': 0, 'ondisk_log_size': 8508, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29527792, 'num_objects': 3605, 'num_object_clones': 0, 'num_object_copies': 14420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8505, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8589", 'reported_seq': 8887, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.955297+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:21.955297+0000', 'last_peered': '2024-09-10T20:26:21.955297+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:21.955297+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8589, 'log_dups_size': 0, 'ondisk_log_size': 8589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29293136, 'num_objects': 3576, 'num_object_clones': 0, 'num_object_copies': 14304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8588, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1716}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8397", 'reported_seq': 8580, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:23.895748+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:23.895748+0000', 'last_peered': '2024-09-10T20:26:23.895748+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:23.895748+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8397, 'log_dups_size': 0, 'ondisk_log_size': 8397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28129872, 'num_objects': 3434, 'num_object_clones': 0, 'num_object_copies': 13736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8396, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8520", 'reported_seq': 8789, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.806658+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:21.806658+0000', 'last_peered': '2024-09-10T20:26:21.806658+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:21.806658+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8520, 'log_dups_size': 0, 'ondisk_log_size': 8520, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29275296, 'num_objects': 3574, 'num_object_clones': 0, 'num_object_copies': 14296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8518, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8757", 'reported_seq': 8962, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:23.986206+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:23.986206+0000', 'last_peered': '2024-09-10T20:26:23.986206+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:23.986206+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8757, 'log_dups_size': 0, 'ondisk_log_size': 8757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30241952, 'num_objects': 3692, 'num_object_clones': 0, 'num_object_copies': 14768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8755, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1784}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:25.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8530", 'reported_seq': 11325, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:21.356351+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:21.356351+0000', 'last_peered': '2024-09-10T20:26:21.356351+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:21.356351+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8530, 'log_dups_size': 0, 'ondisk_log_size': 8530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29447328, 'num_objects': 3595, 'num_object_clones': 0, 'num_object_copies': 14380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 625, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8528, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1358, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:25.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:26.695 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21411/229896 objects degraded (9.313%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57474,"data_bytes":470791176,"bytes_used":2794676224,"bytes_avail":383752380416,"bytes_total":386547056640,"degraded_objects":21411,"degraded_total":229896,"degraded_ratio":0.093133416849357969,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:26.719 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:26.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:27.369 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-10T20:26:26.030187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":470791176,"num_objects":57474,"num_object_clones":0,"num_object_copies":229896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116871,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136997,"ondisk_log_size":136997,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729176,"kb_used_data":921720,"kb_used_omap":14,"kb_used_meta":1807345,"kb_avail":374758184,"statfs":{"total":386547056640,"available":383752380416,"internally_reserved":0,"allocated":943841280,"data_stored":942989036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1850721846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,13,19,19,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20430848,"num_objects":-2494,"num_object_clones":0,"num_object_copies":-9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005338"},"pg_stats":[{"pgid":"1.f","version":"40'8519","reported_seq":8801,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.903179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:21.903179+0000","last_peered":"2024-09-10T20:26:21.903179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:21.903179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8519,"log_dups_size":0,"ondisk_log_size":8519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29515776,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8683","reported_seq":8815,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:24.822066+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:24.822066+0000","last_peered":"2024-09-10T20:26:24.822066+0000","last_clean":"2024-09-10T20:26:24.822066+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:24.822066+0000","last_undegraded":"2024-09-10T20:26:24.822066+0000","last_fullsized":"2024-09-10T20:26:24.822066+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8683,"log_dups_size":0,"ondisk_log_size":8683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29237248,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8552","reported_seq":8755,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.864593+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:23.864593+0000","last_peered":"2024-09-10T20:26:23.864593+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:23.864593+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8552,"log_dups_size":0,"ondisk_log_size":8552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29391464,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8551,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8760","reported_seq":12929,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:24.877913+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:24.877913+0000","last_peered":"2024-09-10T20:26:24.877913+0000","last_clean":"2024-09-10T20:26:24.877913+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:24.877913+0000","last_undegraded":"2024-09-10T20:26:24.877913+0000","last_fullsized":"2024-09-10T20:26:24.877913+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8760,"log_dups_size":0,"ondisk_log_size":8760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29868032,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8597","reported_seq":8917,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:22.035972+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:22.035972+0000","last_peered":"2024-09-10T20:26:22.035972+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:22.035972+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29258912,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8496","reported_seq":8718,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.912069+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:21.912069+0000","last_peered":"2024-09-10T20:26:21.912069+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:21.912069+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29507584,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8496,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8500","reported_seq":8703,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.336420+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:21.336420+0000","last_peered":"2024-09-10T20:26:21.336420+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:21.336420+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1677}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8636","reported_seq":8801,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.353449+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:21.353449+0000","last_peered":"2024-09-10T20:26:21.353449+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:21.353449+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955232,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8427","reported_seq":12397,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:22.013591+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:22.013591+0000","last_peered":"2024-09-10T20:26:22.013591+0000","last_clean":"2024-09-10T20:26:22.013591+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:22.013591+0000","last_undegraded":"2024-09-10T20:26:22.013591+0000","last_fullsized":"2024-09-10T20:26:22.013591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8427,"log_dups_size":0,"ondisk_log_size":8427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29276752,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8526","reported_seq":12494,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:23.907053+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:23.907053+0000","last_peered":"2024-09-10T20:26:23.907053+0000","last_clean":"2024-09-10T20:26:23.907053+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:23.907053+0000","last_undegraded":"2024-09-10T20:26:23.907053+0000","last_fullsized":"2024-09-10T20:26:23.907053+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8508","reported_seq":8781,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.954373+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:21.954373+0000","last_peered":"2024-09-10T20:26:21.954373+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:21.954373+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8508,"log_dups_size":0,"ondisk_log_size":8508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29527792,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8505,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8589","reported_seq":8887,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.955297+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:21.955297+0000","last_peered":"2024-09-10T20:26:21.955297+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:21.955297+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29293136,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8397","reported_seq":8580,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.895748+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:23.895748+0000","last_peered":"2024-09-10T20:26:23.895748+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:23.895748+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28129872,"num_objects":3434,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8520","reported_seq":8789,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.806658+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:21.806658+0000","last_peered":"2024-09-10T20:26:21.806658+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:21.806658+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29275296,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8757","reported_seq":8962,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:23.986206+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:23.986206+0000","last_peered":"2024-09-10T20:26:23.986206+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:23.986206+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30241952,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8530","reported_seq":11325,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:21.356351+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:21.356351+0000","last_peered":"2024-09-10T20:26:21.356351+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:21.356351+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447328,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1358,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":470791176,"num_objects":57474,"num_object_clones":0,"num_object_copies":229896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116871,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":940351488,"data_stored":940351488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136997,"ondisk_log_size":136997,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704620,"kb_used_data":231444,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93667220,"statfs":{"total":96636764160,"available":95915233280,"internally_reserved":0,"allocated":236998656,"data_stored":236785595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689808,"kb_used_data":229608,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93682032,"statfs":{"total":96636764160,"available":95930400768,"internally_reserved":0,"allocated":235118592,"data_stored":234905531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629380,"kb_used_data":228404,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93742460,"statfs":{"total":96636764160,"available":95992279040,"internally_reserved":0,"allocated":233885696,"data_stored":233672635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":410578642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,2,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705368,"kb_used_data":232264,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93666472,"statfs":{"total":96636764160,"available":95914467328,"internally_reserved":0,"allocated":237838336,"data_stored":237625275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236740608,"data_stored":236740608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233459712,"data_stored":233459712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234061824,"data_stored":234061824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236089344,"data_stored":236089344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:30.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:30.999 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:31.000 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:31.017 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-10T20:26:30.031448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":461001736,"num_objects":56279,"num_object_clones":0,"num_object_copies":225116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56279,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118066,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138192,"ondisk_log_size":138192,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676784,"kb_used_data":902620,"kb_used_omap":14,"kb_used_meta":1774065,"kb_avail":374810576,"statfs":{"total":386547056640,"available":383806029824,"internally_reserved":0,"allocated":924282880,"data_stored":923430636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1816643126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,17,16,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23232512,"num_objects":-2836,"num_object_clones":0,"num_object_copies":-11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004668"},"pg_stats":[{"pgid":"1.f","version":"40'8590","reported_seq":8871,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:27.065396+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:27.065396+0000","last_peered":"2024-09-10T20:26:27.065396+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:27.065396+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8590,"log_dups_size":0,"ondisk_log_size":8590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28934144,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8757","reported_seq":8888,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:29.947397+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:29.947397+0000","last_peered":"2024-09-10T20:26:29.947397+0000","last_clean":"2024-09-10T20:26:29.947397+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:29.947397+0000","last_undegraded":"2024-09-10T20:26:29.947397+0000","last_fullsized":"2024-09-10T20:26:29.947397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28631040,"num_objects":3495,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8618","reported_seq":8820,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.974817+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:28.974817+0000","last_peered":"2024-09-10T20:26:28.974817+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:28.974817+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28850792,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8617,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8828","reported_seq":12997,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:29.800727+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:29.800727+0000","last_peered":"2024-09-10T20:26:29.800727+0000","last_clean":"2024-09-10T20:26:29.800727+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:29.800727+0000","last_undegraded":"2024-09-10T20:26:29.800727+0000","last_fullsized":"2024-09-10T20:26:29.800727+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8828,"log_dups_size":0,"ondisk_log_size":8828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8675","reported_seq":8995,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.976614+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:26.976614+0000","last_peered":"2024-09-10T20:26:26.976614+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:26.976614+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8675,"log_dups_size":0,"ondisk_log_size":8675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28619936,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8590","reported_seq":8812,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.864856+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:26.864856+0000","last_peered":"2024-09-10T20:26:26.864856+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:26.864856+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8590,"log_dups_size":0,"ondisk_log_size":8590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8574","reported_seq":8777,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.296758+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:26.296758+0000","last_peered":"2024-09-10T20:26:26.296758+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:26.296758+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8574,"log_dups_size":0,"ondisk_log_size":8574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1603}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8702","reported_seq":8870,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.390491+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:26.390491+0000","last_peered":"2024-09-10T20:26:26.390491+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:26.390491+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8702,"log_dups_size":0,"ondisk_log_size":8702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29414560,"num_objects":3591,"num_object_clones":0,"num_object_copies":14364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8700,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8513","reported_seq":12485,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:27.022342+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:27.022342+0000","last_peered":"2024-09-10T20:26:27.022342+0000","last_clean":"2024-09-10T20:26:27.022342+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:27.022342+0000","last_undegraded":"2024-09-10T20:26:27.022342+0000","last_fullsized":"2024-09-10T20:26:27.022342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28572240,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8592","reported_seq":12562,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:28.974646+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:28.974646+0000","last_peered":"2024-09-10T20:26:28.974646+0000","last_clean":"2024-09-10T20:26:28.974646+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:28.974646+0000","last_undegraded":"2024-09-10T20:26:28.974646+0000","last_fullsized":"2024-09-10T20:26:28.974646+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29081600,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1230,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8584","reported_seq":8857,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:27.021592+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:27.021592+0000","last_peered":"2024-09-10T20:26:27.021592+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:27.021592+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28905200,"num_objects":3529,"num_object_clones":0,"num_object_copies":14116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8663","reported_seq":8961,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.971048+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:26.971048+0000","last_peered":"2024-09-10T20:26:26.971048+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:26.971048+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28686928,"num_objects":3502,"num_object_clones":0,"num_object_copies":14008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8472","reported_seq":8655,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.831906+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:28.831906+0000","last_peered":"2024-09-10T20:26:28.831906+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:28.831906+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27515472,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1456}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8598","reported_seq":8866,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.925603+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:26.925603+0000","last_peered":"2024-09-10T20:26:26.925603+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:26.925603+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8598,"log_dups_size":0,"ondisk_log_size":8598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8827","reported_seq":9032,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.520617+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:28.520617+0000","last_peered":"2024-09-10T20:26:28.520617+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:28.520617+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8827,"log_dups_size":0,"ondisk_log_size":8827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668512,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8825,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8609","reported_seq":11551,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.395465+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:26.395465+0000","last_peered":"2024-09-10T20:26:26.395465+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:26.395465+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28800160,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":461001736,"num_objects":56279,"num_object_clones":0,"num_object_copies":225116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56279,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118066,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":920952832,"data_stored":920952832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138192,"ondisk_log_size":138192,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687380,"kb_used_data":226692,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93684460,"statfs":{"total":96636764160,"available":95932887040,"internally_reserved":0,"allocated":232132608,"data_stored":231919547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689236,"kb_used_data":224884,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93682604,"statfs":{"total":96636764160,"available":95930986496,"internally_reserved":0,"allocated":230281216,"data_stored":230068155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,11,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612200,"kb_used_data":223704,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93759640,"statfs":{"total":96636764160,"available":96009871360,"internally_reserved":0,"allocated":229072896,"data_stored":228859835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215193,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687968,"kb_used_data":227340,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93683872,"statfs":{"total":96636764160,"available":95932284928,"internally_reserved":0,"allocated":232796160,"data_stored":232583099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231841792,"data_stored":231841792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228589568,"data_stored":228589568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229261312,"data_stored":229261312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231260160,"data_stored":231260160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:31.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:31.628 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:31.628 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:31.663 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-10T20:26:30.031448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":461001736,"num_objects":56279,"num_object_clones":0,"num_object_copies":225116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56279,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118066,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138192,"ondisk_log_size":138192,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676784,"kb_used_data":902620,"kb_used_omap":14,"kb_used_meta":1774065,"kb_avail":374810576,"statfs":{"total":386547056640,"available":383806029824,"internally_reserved":0,"allocated":924282880,"data_stored":923430636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1816643126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,17,16,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23232512,"num_objects":-2836,"num_object_clones":0,"num_object_copies":-11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004668"},"pg_stats":[{"pgid":"1.f","version":"40'8590","reported_seq":8871,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:27.065396+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:27.065396+0000","last_peered":"2024-09-10T20:26:27.065396+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:27.065396+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8590,"log_dups_size":0,"ondisk_log_size":8590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28934144,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8757","reported_seq":8888,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:29.947397+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:29.947397+0000","last_peered":"2024-09-10T20:26:29.947397+0000","last_clean":"2024-09-10T20:26:29.947397+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:29.947397+0000","last_undegraded":"2024-09-10T20:26:29.947397+0000","last_fullsized":"2024-09-10T20:26:29.947397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28631040,"num_objects":3495,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8618","reported_seq":8820,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.974817+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:28.974817+0000","last_peered":"2024-09-10T20:26:28.974817+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:28.974817+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28850792,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8617,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8828","reported_seq":12997,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:29.800727+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:29.800727+0000","last_peered":"2024-09-10T20:26:29.800727+0000","last_clean":"2024-09-10T20:26:29.800727+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:29.800727+0000","last_undegraded":"2024-09-10T20:26:29.800727+0000","last_fullsized":"2024-09-10T20:26:29.800727+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8828,"log_dups_size":0,"ondisk_log_size":8828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8675","reported_seq":8995,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.976614+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:26.976614+0000","last_peered":"2024-09-10T20:26:26.976614+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:26.976614+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8675,"log_dups_size":0,"ondisk_log_size":8675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28619936,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8590","reported_seq":8812,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.864856+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:26.864856+0000","last_peered":"2024-09-10T20:26:26.864856+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:26.864856+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8590,"log_dups_size":0,"ondisk_log_size":8590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8574","reported_seq":8777,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.296758+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:26.296758+0000","last_peered":"2024-09-10T20:26:26.296758+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:26.296758+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8574,"log_dups_size":0,"ondisk_log_size":8574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1603}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8702","reported_seq":8870,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.390491+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:26.390491+0000","last_peered":"2024-09-10T20:26:26.390491+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:26.390491+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8702,"log_dups_size":0,"ondisk_log_size":8702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29414560,"num_objects":3591,"num_object_clones":0,"num_object_copies":14364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8700,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8513","reported_seq":12485,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:27.022342+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:27.022342+0000","last_peered":"2024-09-10T20:26:27.022342+0000","last_clean":"2024-09-10T20:26:27.022342+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:27.022342+0000","last_undegraded":"2024-09-10T20:26:27.022342+0000","last_fullsized":"2024-09-10T20:26:27.022342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28572240,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8592","reported_seq":12562,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:28.974646+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:28.974646+0000","last_peered":"2024-09-10T20:26:28.974646+0000","last_clean":"2024-09-10T20:26:28.974646+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:28.974646+0000","last_undegraded":"2024-09-10T20:26:28.974646+0000","last_fullsized":"2024-09-10T20:26:28.974646+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29081600,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1230,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8584","reported_seq":8857,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:27.021592+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:27.021592+0000","last_peered":"2024-09-10T20:26:27.021592+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:27.021592+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28905200,"num_objects":3529,"num_object_clones":0,"num_object_copies":14116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8663","reported_seq":8961,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.971048+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:26.971048+0000","last_peered":"2024-09-10T20:26:26.971048+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:26.971048+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28686928,"num_objects":3502,"num_object_clones":0,"num_object_copies":14008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8472","reported_seq":8655,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.831906+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:28.831906+0000","last_peered":"2024-09-10T20:26:28.831906+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:28.831906+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27515472,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1456}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8598","reported_seq":8866,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.925603+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:26.925603+0000","last_peered":"2024-09-10T20:26:26.925603+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:26.925603+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8598,"log_dups_size":0,"ondisk_log_size":8598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8827","reported_seq":9032,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.520617+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:28.520617+0000","last_peered":"2024-09-10T20:26:28.520617+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:28.520617+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8827,"log_dups_size":0,"ondisk_log_size":8827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668512,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8825,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8609","reported_seq":11551,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.395465+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:26.395465+0000","last_peered":"2024-09-10T20:26:26.395465+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:26.395465+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28800160,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":461001736,"num_objects":56279,"num_object_clones":0,"num_object_copies":225116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56279,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118066,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":920952832,"data_stored":920952832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138192,"ondisk_log_size":138192,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687380,"kb_used_data":226692,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93684460,"statfs":{"total":96636764160,"available":95932887040,"internally_reserved":0,"allocated":232132608,"data_stored":231919547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689236,"kb_used_data":224884,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93682604,"statfs":{"total":96636764160,"available":95930986496,"internally_reserved":0,"allocated":230281216,"data_stored":230068155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,11,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612200,"kb_used_data":223704,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93759640,"statfs":{"total":96636764160,"available":96009871360,"internally_reserved":0,"allocated":229072896,"data_stored":228859835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215193,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687968,"kb_used_data":227340,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93683872,"statfs":{"total":96636764160,"available":95932284928,"internally_reserved":0,"allocated":232796160,"data_stored":232583099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231841792,"data_stored":231841792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228589568,"data_stored":228589568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229261312,"data_stored":229261312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231260160,"data_stored":231260160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:31.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:31.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8590", 'reported_seq': 8871, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:27.065396+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:27.065396+0000', 'last_peered': '2024-09-10T20:26:27.065396+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:27.065396+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8590, 'log_dups_size': 0, 'ondisk_log_size': 8590, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28934144, 'num_objects': 3532, 'num_object_clones': 0, 'num_object_copies': 14128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8590, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1677}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8618", 'reported_seq': 8820, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:28.974817+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:28.974817+0000', 'last_peered': '2024-09-10T20:26:28.974817+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:28.974817+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8618, 'log_dups_size': 0, 'ondisk_log_size': 8618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28850792, 'num_objects': 3523, 'num_object_clones': 0, 'num_object_copies': 14092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3523, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8617, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1615}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8675", 'reported_seq': 8995, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:26.976614+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:26.976614+0000', 'last_peered': '2024-09-10T20:26:26.976614+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:26.976614+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8675, 'log_dups_size': 0, 'ondisk_log_size': 8675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28619936, 'num_objects': 3494, 'num_object_clones': 0, 'num_object_copies': 13976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8673, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1639}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8590", 'reported_seq': 8812, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:26.864856+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:26.864856+0000', 'last_peered': '2024-09-10T20:26:26.864856+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:26.864856+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8590, 'log_dups_size': 0, 'ondisk_log_size': 8590, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28737536, 'num_objects': 3508, 'num_object_clones': 0, 'num_object_copies': 14032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8590, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1651}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8574", 'reported_seq': 8777, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:26.296758+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:26.296758+0000', 'last_peered': '2024-09-10T20:26:26.296758+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:26.296758+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8574, 'log_dups_size': 0, 'ondisk_log_size': 8574, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28636320, 'num_objects': 3496, 'num_object_clones': 0, 'num_object_copies': 13984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8572, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1603}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:31.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8702", 'reported_seq': 8870, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:26.390491+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:26.390491+0000', 'last_peered': '2024-09-10T20:26:26.390491+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:26.390491+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8702, 'log_dups_size': 0, 'ondisk_log_size': 8702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29414560, 'num_objects': 3591, 'num_object_clones': 0, 'num_object_copies': 14364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8700, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8584", 'reported_seq': 8857, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:27.021592+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:27.021592+0000', 'last_peered': '2024-09-10T20:26:27.021592+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:27.021592+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8584, 'log_dups_size': 0, 'ondisk_log_size': 8584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28905200, 'num_objects': 3529, 'num_object_clones': 0, 'num_object_copies': 14116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8581, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1673}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8663", 'reported_seq': 8961, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:26.971048+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:26.971048+0000', 'last_peered': '2024-09-10T20:26:26.971048+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:26.971048+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8663, 'log_dups_size': 0, 'ondisk_log_size': 8663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28686928, 'num_objects': 3502, 'num_object_clones': 0, 'num_object_copies': 14008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8662, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8472", 'reported_seq': 8655, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:28.831906+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:28.831906+0000', 'last_peered': '2024-09-10T20:26:28.831906+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:28.831906+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8472, 'log_dups_size': 0, 'ondisk_log_size': 8472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27515472, 'num_objects': 3359, 'num_object_clones': 0, 'num_object_copies': 13436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8471, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1456}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8598", 'reported_seq': 8866, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:26.925603+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:26.925603+0000', 'last_peered': '2024-09-10T20:26:26.925603+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:26.925603+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8598, 'log_dups_size': 0, 'ondisk_log_size': 8598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28636320, 'num_objects': 3496, 'num_object_clones': 0, 'num_object_copies': 13984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8596, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1621}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8827", 'reported_seq': 9032, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:28.520617+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:28.520617+0000', 'last_peered': '2024-09-10T20:26:28.520617+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:28.520617+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8827, 'log_dups_size': 0, 'ondisk_log_size': 8827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29668512, 'num_objects': 3622, 'num_object_clones': 0, 'num_object_copies': 14488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8825, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1714}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:31.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:31.667 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8609", 'reported_seq': 11551, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:26.395465+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:26.395465+0000', 'last_peered': '2024-09-10T20:26:26.395465+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:26.395465+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8609, 'log_dups_size': 0, 'ondisk_log_size': 8609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28800160, 'num_objects': 3516, 'num_object_clones': 0, 'num_object_copies': 14064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 551, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8607, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1432, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2965}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:31.667 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:32.428 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21275/224196 objects degraded (9.489%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":481,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56049,"data_bytes":459117576,"bytes_used":2740518912,"bytes_avail":383806537728,"bytes_total":386547056640,"degraded_objects":21275,"degraded_total":224196,"degraded_ratio":0.094894645756391713,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":216},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:32.450 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:32.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:33.015 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:33.016 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:33.040 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-10T20:26:32.032128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459117576,"num_objects":56049,"num_object_clones":0,"num_object_copies":224196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56049,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118296,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2401,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138422,"ondisk_log_size":138422,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676288,"kb_used_data":897956,"kb_used_omap":14,"kb_used_meta":1778225,"kb_avail":374811072,"statfs":{"total":386547056640,"available":383806537728,"internally_reserved":0,"allocated":919506944,"data_stored":918654700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1820902966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,15,23,14,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21258240,"num_objects":-2595,"num_object_clones":0,"num_object_copies":-10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004495"},"pg_stats":[{"pgid":"1.f","version":"40'8590","reported_seq":8871,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:27.065396+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:27.065396+0000","last_peered":"2024-09-10T20:26:27.065396+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:27.065396+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8590,"log_dups_size":0,"ondisk_log_size":8590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28934144,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8757","reported_seq":8888,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:29.947397+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:29.947397+0000","last_peered":"2024-09-10T20:26:29.947397+0000","last_clean":"2024-09-10T20:26:29.947397+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:29.947397+0000","last_undegraded":"2024-09-10T20:26:29.947397+0000","last_fullsized":"2024-09-10T20:26:29.947397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28631040,"num_objects":3495,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8618","reported_seq":8820,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.974817+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:28.974817+0000","last_peered":"2024-09-10T20:26:28.974817+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:28.974817+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28850792,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8617,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8828","reported_seq":12997,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:29.800727+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:29.800727+0000","last_peered":"2024-09-10T20:26:29.800727+0000","last_clean":"2024-09-10T20:26:29.800727+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:29.800727+0000","last_undegraded":"2024-09-10T20:26:29.800727+0000","last_fullsized":"2024-09-10T20:26:29.800727+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8828,"log_dups_size":0,"ondisk_log_size":8828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8675","reported_seq":8995,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.976614+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:26.976614+0000","last_peered":"2024-09-10T20:26:26.976614+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:26.976614+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8675,"log_dups_size":0,"ondisk_log_size":8675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28619936,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8590","reported_seq":8812,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.864856+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:26.864856+0000","last_peered":"2024-09-10T20:26:26.864856+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:26.864856+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8590,"log_dups_size":0,"ondisk_log_size":8590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8637","reported_seq":8840,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.430931+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:31.430931+0000","last_peered":"2024-09-10T20:26:31.430931+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:31.430931+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28120224,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8786","reported_seq":8954,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.406946+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:31.406946+0000","last_peered":"2024-09-10T20:26:31.406946+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:31.406946+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28726432,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8513","reported_seq":12485,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:27.022342+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:27.022342+0000","last_peered":"2024-09-10T20:26:27.022342+0000","last_clean":"2024-09-10T20:26:27.022342+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:27.022342+0000","last_undegraded":"2024-09-10T20:26:27.022342+0000","last_fullsized":"2024-09-10T20:26:27.022342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28572240,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8592","reported_seq":12562,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:28.974646+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:28.974646+0000","last_peered":"2024-09-10T20:26:28.974646+0000","last_clean":"2024-09-10T20:26:28.974646+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:28.974646+0000","last_undegraded":"2024-09-10T20:26:28.974646+0000","last_fullsized":"2024-09-10T20:26:28.974646+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29081600,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1230,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8584","reported_seq":8857,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:27.021592+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:27.021592+0000","last_peered":"2024-09-10T20:26:27.021592+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:27.021592+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28905200,"num_objects":3529,"num_object_clones":0,"num_object_copies":14116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8663","reported_seq":8961,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.971048+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:26.971048+0000","last_peered":"2024-09-10T20:26:26.971048+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:26.971048+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28686928,"num_objects":3502,"num_object_clones":0,"num_object_copies":14008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8472","reported_seq":8655,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.831906+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:28.831906+0000","last_peered":"2024-09-10T20:26:28.831906+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:28.831906+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27515472,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1456}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8598","reported_seq":8866,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:26.925603+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:26.925603+0000","last_peered":"2024-09-10T20:26:26.925603+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:26.925603+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8598,"log_dups_size":0,"ondisk_log_size":8598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8827","reported_seq":9032,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:28.520617+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:28.520617+0000","last_peered":"2024-09-10T20:26:28.520617+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:28.520617+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8827,"log_dups_size":0,"ondisk_log_size":8827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668512,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8825,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8692","reported_seq":11759,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.484433+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:31.484433+0000","last_peered":"2024-09-10T20:26:31.484433+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:31.484433+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8692,"log_dups_size":0,"ondisk_log_size":8692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28120224,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459117576,"num_objects":56049,"num_object_clones":0,"num_object_copies":224196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56049,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118296,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2401,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":916041728,"data_stored":916041728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138422,"ondisk_log_size":138422,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687380,"kb_used_data":226692,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93684460,"statfs":{"total":96636764160,"available":95932887040,"internally_reserved":0,"allocated":232132608,"data_stored":231919547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689236,"kb_used_data":224884,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93682604,"statfs":{"total":96636764160,"available":95930986496,"internally_reserved":0,"allocated":230281216,"data_stored":230068155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,11,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612200,"kb_used_data":223704,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93759640,"statfs":{"total":96636764160,"available":96009871360,"internally_reserved":0,"allocated":229072896,"data_stored":228859835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687472,"kb_used_data":222676,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93684368,"statfs":{"total":96636764160,"available":95932792832,"internally_reserved":0,"allocated":228020224,"data_stored":227807163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226930688,"data_stored":226930688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228589568,"data_stored":228589568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229261312,"data_stored":229261312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231260160,"data_stored":231260160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:36.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:36.597 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:36.608 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-10T20:26:36.033610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451236872,"num_objects":55087,"num_object_clones":0,"num_object_copies":220348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119258,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2401,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139384,"ondisk_log_size":139384,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674920,"kb_used_data":884100,"kb_used_omap":14,"kb_used_meta":1790705,"kb_avail":374812440,"statfs":{"total":386547056640,"available":383807938560,"internally_reserved":0,"allocated":905318400,"data_stored":904466156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1833682486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,1,14,20,16,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20684800,"num_objects":-2525,"num_object_clones":0,"num_object_copies":-10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004271"},"pg_stats":[{"pgid":"1.f","version":"40'8666","reported_seq":8948,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.976506+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:31.976506+0000","last_peered":"2024-09-10T20:26:31.976506+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:31.976506+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8666,"log_dups_size":0,"ondisk_log_size":8666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8666,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8835","reported_seq":8967,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.854524+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:34.854524+0000","last_peered":"2024-09-10T20:26:34.854524+0000","last_clean":"2024-09-10T20:26:34.854524+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:34.854524+0000","last_undegraded":"2024-09-10T20:26:34.854524+0000","last_fullsized":"2024-09-10T20:26:34.854524+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8835,"log_dups_size":0,"ondisk_log_size":8835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27992064,"num_objects":3417,"num_object_clones":0,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8685","reported_seq":8888,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:33.995723+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:33.995723+0000","last_peered":"2024-09-10T20:26:33.995723+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:33.995723+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301928,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8684,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8917","reported_seq":13084,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.952641+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:34.952641+0000","last_peered":"2024-09-10T20:26:34.952641+0000","last_clean":"2024-09-10T20:26:34.952641+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:34.952641+0000","last_undegraded":"2024-09-10T20:26:34.952641+0000","last_fullsized":"2024-09-10T20:26:34.952641+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28581888,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8755","reported_seq":9075,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.864380+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:31.864380+0000","last_peered":"2024-09-10T20:26:31.864380+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:31.864380+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27964576,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8753,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8658","reported_seq":8880,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.906850+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:31.906850+0000","last_peered":"2024-09-10T20:26:31.906850+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:31.906850+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8658,"log_dups_size":0,"ondisk_log_size":8658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8658,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8637","reported_seq":8840,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.430931+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:31.430931+0000","last_peered":"2024-09-10T20:26:31.430931+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:31.430931+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28120224,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8786","reported_seq":8954,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.406946+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:31.406946+0000","last_peered":"2024-09-10T20:26:31.406946+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:31.406946+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28726432,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8591","reported_seq":12563,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:31.822278+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:31.822278+0000","last_peered":"2024-09-10T20:26:31.822278+0000","last_clean":"2024-09-10T20:26:31.822278+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:31.822278+0000","last_undegraded":"2024-09-10T20:26:31.822278+0000","last_fullsized":"2024-09-10T20:26:31.822278+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27933264,"num_objects":3410,"num_object_clones":0,"num_object_copies":13640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5144,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8651","reported_seq":12621,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.131045+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:34.131045+0000","last_peered":"2024-09-10T20:26:34.131045+0000","last_clean":"2024-09-10T20:26:34.131045+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:34.131045+0000","last_undegraded":"2024-09-10T20:26:34.131045+0000","last_fullsized":"2024-09-10T20:26:34.131045+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8651,"log_dups_size":0,"ondisk_log_size":8651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8653","reported_seq":8926,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.987790+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:31.987790+0000","last_peered":"2024-09-10T20:26:31.987790+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:31.987790+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8653,"log_dups_size":0,"ondisk_log_size":8653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28339952,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8730","reported_seq":9028,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:32.046570+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:32.046570+0000","last_peered":"2024-09-10T20:26:32.046570+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:32.046570+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28138064,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1575}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8537","reported_seq":8720,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:34.030182+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:34.030182+0000","last_peered":"2024-09-10T20:26:34.030182+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:34.030182+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26982992,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1391}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8674","reported_seq":8943,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.930630+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:31.930630+0000","last_peered":"2024-09-10T20:26:31.930630+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:31.930630+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8674,"log_dups_size":0,"ondisk_log_size":8674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013728,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8672,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8917","reported_seq":9122,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:34.130836+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:34.130836+0000","last_peered":"2024-09-10T20:26:34.130836+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:34.130836+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8915,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8692","reported_seq":11759,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.484433+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:31.484433+0000","last_peered":"2024-09-10T20:26:31.484433+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:31.484433+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8692,"log_dups_size":0,"ondisk_log_size":8692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28120224,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451236872,"num_objects":55087,"num_object_clones":0,"num_object_copies":220348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119258,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2401,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":901685248,"data_stored":901685248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139384,"ondisk_log_size":139384,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686884,"kb_used_data":222012,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93684956,"statfs":{"total":96636764160,"available":95933394944,"internally_reserved":0,"allocated":227340288,"data_stored":227127227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,7,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688676,"kb_used_data":220180,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93683164,"statfs":{"total":96636764160,"available":95931559936,"internally_reserved":0,"allocated":225464320,"data_stored":225251259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,2,4,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611888,"kb_used_data":219232,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93759952,"statfs":{"total":96636764160,"available":96010190848,"internally_reserved":0,"allocated":224493568,"data_stored":224280507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,7,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687472,"kb_used_data":222676,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93684368,"statfs":{"total":96636764160,"available":95932792832,"internally_reserved":0,"allocated":228020224,"data_stored":227807163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226930688,"data_stored":226930688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223866880,"data_stored":223866880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224473088,"data_stored":224473088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":226414592,"data_stored":226414592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:36.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:37.244 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:37.263 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-10T20:26:36.033610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451236872,"num_objects":55087,"num_object_clones":0,"num_object_copies":220348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119258,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2401,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139384,"ondisk_log_size":139384,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674920,"kb_used_data":884100,"kb_used_omap":14,"kb_used_meta":1790705,"kb_avail":374812440,"statfs":{"total":386547056640,"available":383807938560,"internally_reserved":0,"allocated":905318400,"data_stored":904466156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1833682486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,1,14,20,16,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20684800,"num_objects":-2525,"num_object_clones":0,"num_object_copies":-10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004271"},"pg_stats":[{"pgid":"1.f","version":"40'8666","reported_seq":8948,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.976506+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:31.976506+0000","last_peered":"2024-09-10T20:26:31.976506+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:31.976506+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8666,"log_dups_size":0,"ondisk_log_size":8666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8666,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8835","reported_seq":8967,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.854524+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:34.854524+0000","last_peered":"2024-09-10T20:26:34.854524+0000","last_clean":"2024-09-10T20:26:34.854524+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:34.854524+0000","last_undegraded":"2024-09-10T20:26:34.854524+0000","last_fullsized":"2024-09-10T20:26:34.854524+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8835,"log_dups_size":0,"ondisk_log_size":8835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27992064,"num_objects":3417,"num_object_clones":0,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8685","reported_seq":8888,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:33.995723+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:33.995723+0000","last_peered":"2024-09-10T20:26:33.995723+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:33.995723+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301928,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8684,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8917","reported_seq":13084,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.952641+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:34.952641+0000","last_peered":"2024-09-10T20:26:34.952641+0000","last_clean":"2024-09-10T20:26:34.952641+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:34.952641+0000","last_undegraded":"2024-09-10T20:26:34.952641+0000","last_fullsized":"2024-09-10T20:26:34.952641+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28581888,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8755","reported_seq":9075,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.864380+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:31.864380+0000","last_peered":"2024-09-10T20:26:31.864380+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:31.864380+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27964576,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8753,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8658","reported_seq":8880,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.906850+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:31.906850+0000","last_peered":"2024-09-10T20:26:31.906850+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:31.906850+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8658,"log_dups_size":0,"ondisk_log_size":8658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8658,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8637","reported_seq":8840,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.430931+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:31.430931+0000","last_peered":"2024-09-10T20:26:31.430931+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:31.430931+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28120224,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8786","reported_seq":8954,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.406946+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:31.406946+0000","last_peered":"2024-09-10T20:26:31.406946+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:31.406946+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28726432,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8591","reported_seq":12563,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:31.822278+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:31.822278+0000","last_peered":"2024-09-10T20:26:31.822278+0000","last_clean":"2024-09-10T20:26:31.822278+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:31.822278+0000","last_undegraded":"2024-09-10T20:26:31.822278+0000","last_fullsized":"2024-09-10T20:26:31.822278+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27933264,"num_objects":3410,"num_object_clones":0,"num_object_copies":13640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5144,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8651","reported_seq":12621,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.131045+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:34.131045+0000","last_peered":"2024-09-10T20:26:34.131045+0000","last_clean":"2024-09-10T20:26:34.131045+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:34.131045+0000","last_undegraded":"2024-09-10T20:26:34.131045+0000","last_fullsized":"2024-09-10T20:26:34.131045+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8651,"log_dups_size":0,"ondisk_log_size":8651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8653","reported_seq":8926,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.987790+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:31.987790+0000","last_peered":"2024-09-10T20:26:31.987790+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:31.987790+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8653,"log_dups_size":0,"ondisk_log_size":8653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28339952,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8730","reported_seq":9028,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:32.046570+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:32.046570+0000","last_peered":"2024-09-10T20:26:32.046570+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:32.046570+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28138064,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1575}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8537","reported_seq":8720,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:34.030182+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:34.030182+0000","last_peered":"2024-09-10T20:26:34.030182+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:34.030182+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26982992,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1391}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8674","reported_seq":8943,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.930630+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:31.930630+0000","last_peered":"2024-09-10T20:26:31.930630+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:31.930630+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8674,"log_dups_size":0,"ondisk_log_size":8674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013728,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8672,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8917","reported_seq":9122,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:34.130836+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:34.130836+0000","last_peered":"2024-09-10T20:26:34.130836+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:34.130836+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8915,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8692","reported_seq":11759,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:31.484433+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:31.484433+0000","last_peered":"2024-09-10T20:26:31.484433+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:31.484433+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8692,"log_dups_size":0,"ondisk_log_size":8692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28120224,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451236872,"num_objects":55087,"num_object_clones":0,"num_object_copies":220348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119258,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2401,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":901685248,"data_stored":901685248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139384,"ondisk_log_size":139384,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686884,"kb_used_data":222012,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93684956,"statfs":{"total":96636764160,"available":95933394944,"internally_reserved":0,"allocated":227340288,"data_stored":227127227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,7,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688676,"kb_used_data":220180,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93683164,"statfs":{"total":96636764160,"available":95931559936,"internally_reserved":0,"allocated":225464320,"data_stored":225251259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,2,4,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611888,"kb_used_data":219232,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93759952,"statfs":{"total":96636764160,"available":96010190848,"internally_reserved":0,"allocated":224493568,"data_stored":224280507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,7,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687472,"kb_used_data":222676,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93684368,"statfs":{"total":96636764160,"available":95932792832,"internally_reserved":0,"allocated":228020224,"data_stored":227807163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226930688,"data_stored":226930688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223866880,"data_stored":223866880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224473088,"data_stored":224473088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":226414592,"data_stored":226414592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:37.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8666", 'reported_seq': 8948, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.976506+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:31.976506+0000', 'last_peered': '2024-09-10T20:26:31.976506+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:31.976506+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8666, 'log_dups_size': 0, 'ondisk_log_size': 8666, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28311552, 'num_objects': 3456, 'num_object_clones': 0, 'num_object_copies': 13824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8666, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1601}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8685", 'reported_seq': 8888, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:33.995723+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:33.995723+0000', 'last_peered': '2024-09-10T20:26:33.995723+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:33.995723+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8685, 'log_dups_size': 0, 'ondisk_log_size': 8685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28301928, 'num_objects': 3456, 'num_object_clones': 0, 'num_object_copies': 13824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3456, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8684, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1548}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8755", 'reported_seq': 9075, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.864380+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:31.864380+0000', 'last_peered': '2024-09-10T20:26:31.864380+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:31.864380+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8755, 'log_dups_size': 0, 'ondisk_log_size': 8755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27964576, 'num_objects': 3414, 'num_object_clones': 0, 'num_object_copies': 13656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8753, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1559}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8658", 'reported_seq': 8880, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.906850+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:31.906850+0000', 'last_peered': '2024-09-10T20:26:31.906850+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:31.906850+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8658, 'log_dups_size': 0, 'ondisk_log_size': 8658, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28180480, 'num_objects': 3440, 'num_object_clones': 0, 'num_object_copies': 13760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8658, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1583}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:37.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8637", 'reported_seq': 8840, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.430931+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:31.430931+0000', 'last_peered': '2024-09-10T20:26:31.430931+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:31.430931+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8637, 'log_dups_size': 0, 'ondisk_log_size': 8637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28120224, 'num_objects': 3433, 'num_object_clones': 0, 'num_object_copies': 13732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8635, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1540}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8786", 'reported_seq': 8954, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.406946+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:31.406946+0000', 'last_peered': '2024-09-10T20:26:31.406946+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:31.406946+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8786, 'log_dups_size': 0, 'ondisk_log_size': 8786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28726432, 'num_objects': 3507, 'num_object_clones': 0, 'num_object_copies': 14028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8784, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8653", 'reported_seq': 8926, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.987790+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:31.987790+0000', 'last_peered': '2024-09-10T20:26:31.987790+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:31.987790+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8653, 'log_dups_size': 0, 'ondisk_log_size': 8653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28339952, 'num_objects': 3460, 'num_object_clones': 0, 'num_object_copies': 13840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8650, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1604}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8730", 'reported_seq': 9028, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:32.046570+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:32.046570+0000', 'last_peered': '2024-09-10T20:26:32.046570+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:32.046570+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8730, 'log_dups_size': 0, 'ondisk_log_size': 8730, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28138064, 'num_objects': 3435, 'num_object_clones': 0, 'num_object_copies': 13740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8729, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8537", 'reported_seq': 8720, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:34.030182+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:34.030182+0000', 'last_peered': '2024-09-10T20:26:34.030182+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:34.030182+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8537, 'log_dups_size': 0, 'ondisk_log_size': 8537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26982992, 'num_objects': 3294, 'num_object_clones': 0, 'num_object_copies': 13176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8536, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1391}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8674", 'reported_seq': 8943, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.930630+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:31.930630+0000', 'last_peered': '2024-09-10T20:26:31.930630+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:31.930630+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8674, 'log_dups_size': 0, 'ondisk_log_size': 8674, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28013728, 'num_objects': 3420, 'num_object_clones': 0, 'num_object_copies': 13680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8672, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:37.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:37.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8917", 'reported_seq': 9122, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:34.130836+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:34.130836+0000', 'last_peered': '2024-09-10T20:26:34.130836+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:34.130836+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8917, 'log_dups_size': 0, 'ondisk_log_size': 8917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28931232, 'num_objects': 3532, 'num_object_clones': 0, 'num_object_copies': 14128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8915, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:37.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:37.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8692", 'reported_seq': 11759, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:31.484433+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:31.484433+0000', 'last_peered': '2024-09-10T20:26:31.484433+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:31.484433+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8692, 'log_dups_size': 0, 'ondisk_log_size': 8692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28120224, 'num_objects': 3433, 'num_object_clones': 0, 'num_object_copies': 13732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 489, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8690, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1494, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:37.267 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:37.999 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:37.999 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21275/220348 objects degraded (9.655%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":486,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55087,"data_bytes":451236872,"bytes_used":2739118080,"bytes_avail":383807938560,"bytes_total":386547056640,"degraded_objects":21275,"degraded_total":220348,"degraded_ratio":0.096551818033292797,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:38.028 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:38.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:38.678 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:38.678 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:38.704 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-10T20:26:38.034435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":445379592,"num_objects":54372,"num_object_clones":0,"num_object_copies":217488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119973,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2462,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140099,"ondisk_log_size":140099,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2678092,"kb_used_data":874820,"kb_used_omap":14,"kb_used_meta":1803185,"kb_avail":374809268,"statfs":{"total":386547056640,"available":383804690432,"internally_reserved":0,"allocated":895815680,"data_stored":894963436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1846462006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8,0,16,14,12,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25411584,"num_objects":-3102,"num_object_clones":0,"num_object_copies":-12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3102,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004247"},"pg_stats":[{"pgid":"1.f","version":"40'8730","reported_seq":9012,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.215889+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:37.215889+0000","last_peered":"2024-09-10T20:26:37.215889+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:37.215889+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27787264,"num_objects":3392,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8835","reported_seq":8967,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.854524+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:34.854524+0000","last_peered":"2024-09-10T20:26:34.854524+0000","last_clean":"2024-09-10T20:26:34.854524+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:34.854524+0000","last_undegraded":"2024-09-10T20:26:34.854524+0000","last_fullsized":"2024-09-10T20:26:34.854524+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8835,"log_dups_size":0,"ondisk_log_size":8835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27992064,"num_objects":3417,"num_object_clones":0,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8685","reported_seq":8888,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:33.995723+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:33.995723+0000","last_peered":"2024-09-10T20:26:33.995723+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:33.995723+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301928,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8684,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8917","reported_seq":13084,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.952641+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:34.952641+0000","last_peered":"2024-09-10T20:26:34.952641+0000","last_clean":"2024-09-10T20:26:34.952641+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:34.952641+0000","last_undegraded":"2024-09-10T20:26:34.952641+0000","last_fullsized":"2024-09-10T20:26:34.952641+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28581888,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8817","reported_seq":9135,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.943643+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:36.943643+0000","last_peered":"2024-09-10T20:26:36.943643+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:36.943643+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456672,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8742","reported_seq":8964,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.150551+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:37.150551+0000","last_peered":"2024-09-10T20:26:37.150551+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:37.150551+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27492352,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8702","reported_seq":8905,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.281643+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:36.281643+0000","last_peered":"2024-09-10T20:26:36.281643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:36.281643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8702,"log_dups_size":0,"ondisk_log_size":8702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27587744,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8700,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8869","reported_seq":9034,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.425880+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:36.425880+0000","last_peered":"2024-09-10T20:26:36.425880+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:36.425880+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28046496,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8667","reported_seq":12639,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:36.858158+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:36.858158+0000","last_peered":"2024-09-10T20:26:36.858158+0000","last_clean":"2024-09-10T20:26:36.858158+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:36.858158+0000","last_undegraded":"2024-09-10T20:26:36.858158+0000","last_fullsized":"2024-09-10T20:26:36.858158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8667,"log_dups_size":0,"ondisk_log_size":8667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27310672,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8651","reported_seq":12621,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:34.131045+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:34.131045+0000","last_peered":"2024-09-10T20:26:34.131045+0000","last_clean":"2024-09-10T20:26:34.131045+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:34.131045+0000","last_undegraded":"2024-09-10T20:26:34.131045+0000","last_fullsized":"2024-09-10T20:26:34.131045+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8651,"log_dups_size":0,"ondisk_log_size":8651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8716","reported_seq":8989,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.981626+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:36.981626+0000","last_peered":"2024-09-10T20:26:36.981626+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:36.981626+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27823856,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8806","reported_seq":9104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.214619+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:37.214619+0000","last_peered":"2024-09-10T20:26:37.214619+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:37.214619+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27515472,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8805,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8537","reported_seq":8720,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:34.030182+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:34.030182+0000","last_peered":"2024-09-10T20:26:34.030182+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:34.030182+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26982992,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1391}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8744","reported_seq":9013,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.118775+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:37.118775+0000","last_peered":"2024-09-10T20:26:37.118775+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:37.118775+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8744,"log_dups_size":0,"ondisk_log_size":8744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440288,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1475}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8917","reported_seq":9122,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:34.130836+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:34.130836+0000","last_peered":"2024-09-10T20:26:34.130836+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:34.130836+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8915,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8764","reported_seq":11948,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.528297+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:36.528297+0000","last_peered":"2024-09-10T20:26:36.528297+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:36.528297+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530400,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1555,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":445379592,"num_objects":54372,"num_object_clones":0,"num_object_copies":217488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119973,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2462,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":892129280,"data_stored":892129280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140099,"ondisk_log_size":140099,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690496,"kb_used_data":217328,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93681344,"statfs":{"total":96636764160,"available":95929696256,"internally_reserved":0,"allocated":222543872,"data_stored":222330811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688676,"kb_used_data":220180,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93683164,"statfs":{"total":96636764160,"available":95931559936,"internally_reserved":0,"allocated":225464320,"data_stored":225251259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,2,4,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611888,"kb_used_data":219232,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93759952,"statfs":{"total":96636764160,"available":96010190848,"internally_reserved":0,"allocated":224493568,"data_stored":224280507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,7,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215196,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687032,"kb_used_data":218080,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93684808,"statfs":{"total":96636764160,"available":95933243392,"internally_reserved":0,"allocated":223313920,"data_stored":223100859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":222208000,"data_stored":222208000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223866880,"data_stored":223866880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224473088,"data_stored":224473088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221581312,"data_stored":221581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:41.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:42.328 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:42.329 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:42.348 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-10T20:26:42.035869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":440235016,"num_objects":53744,"num_object_clones":0,"num_object_copies":214976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120601,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2598,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140727,"ondisk_log_size":140727,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717428,"kb_used_data":861416,"kb_used_omap":14,"kb_used_meta":1855921,"kb_avail":374769932,"statfs":{"total":386547056640,"available":383764410368,"internally_reserved":0,"allocated":882089984,"data_stored":881237740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1900463670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,2,6,18,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20766720,"num_objects":-2535,"num_object_clones":0,"num_object_copies":-10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2535,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004421"},"pg_stats":[{"pgid":"1.f","version":"40'8730","reported_seq":9012,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.215889+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:37.215889+0000","last_peered":"2024-09-10T20:26:37.215889+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:37.215889+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27787264,"num_objects":3392,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8899","reported_seq":9031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:39.936224+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:39.936224+0000","last_peered":"2024-09-10T20:26:39.936224+0000","last_clean":"2024-09-10T20:26:39.936224+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:39.936224+0000","last_undegraded":"2024-09-10T20:26:39.936224+0000","last_fullsized":"2024-09-10T20:26:39.936224+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8899,"log_dups_size":0,"ondisk_log_size":8899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27467776,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8774","reported_seq":8977,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.075624+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:39.075624+0000","last_peered":"2024-09-10T20:26:39.075624+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:39.075624+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8774,"log_dups_size":0,"ondisk_log_size":8774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27572840,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8773,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8992","reported_seq":13161,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:40.073587+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:40.073587+0000","last_peered":"2024-09-10T20:26:40.073587+0000","last_clean":"2024-09-10T20:26:40.073587+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:40.073587+0000","last_undegraded":"2024-09-10T20:26:40.073587+0000","last_fullsized":"2024-09-10T20:26:40.073587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27967488,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8817","reported_seq":9135,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.943643+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:36.943643+0000","last_peered":"2024-09-10T20:26:36.943643+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:36.943643+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456672,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8742","reported_seq":8964,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.150551+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:37.150551+0000","last_peered":"2024-09-10T20:26:37.150551+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:37.150551+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27492352,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8786","reported_seq":8989,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.529887+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:41.529887+0000","last_peered":"2024-09-10T20:26:41.529887+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:41.529887+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26899616,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8930","reported_seq":9098,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.262817+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:41.262817+0000","last_peered":"2024-09-10T20:26:41.262817+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:41.262817+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27546784,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8667","reported_seq":12639,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:36.858158+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:36.858158+0000","last_peered":"2024-09-10T20:26:36.858158+0000","last_clean":"2024-09-10T20:26:36.858158+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:36.858158+0000","last_undegraded":"2024-09-10T20:26:36.858158+0000","last_fullsized":"2024-09-10T20:26:36.858158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8667,"log_dups_size":0,"ondisk_log_size":8667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27310672,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8716","reported_seq":12686,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:39.173016+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:39.173016+0000","last_peered":"2024-09-10T20:26:39.173016+0000","last_clean":"2024-09-10T20:26:39.173016+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:39.173016+0000","last_undegraded":"2024-09-10T20:26:39.173016+0000","last_fullsized":"2024-09-10T20:26:39.173016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28065792,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8716","reported_seq":8989,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.981626+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:36.981626+0000","last_peered":"2024-09-10T20:26:36.981626+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:36.981626+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27823856,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8806","reported_seq":9104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.214619+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:37.214619+0000","last_peered":"2024-09-10T20:26:37.214619+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:37.214619+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27515472,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8805,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8597","reported_seq":8780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.122713+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:39.122713+0000","last_peered":"2024-09-10T20:26:39.122713+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:39.122713+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491472,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1331}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8744","reported_seq":9013,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.118775+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:37.118775+0000","last_peered":"2024-09-10T20:26:37.118775+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:37.118775+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8744,"log_dups_size":0,"ondisk_log_size":8744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440288,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1475}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8975","reported_seq":9180,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.066458+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:39.066458+0000","last_peered":"2024-09-10T20:26:39.066458+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:39.066458+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8836","reported_seq":12297,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.542561+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:41.542561+0000","last_peered":"2024-09-10T20:26:41.542561+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:41.542561+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26940576,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":440235016,"num_objects":53744,"num_object_clones":0,"num_object_copies":214976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120601,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2598,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":878469120,"data_stored":878469120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140727,"ondisk_log_size":140727,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690496,"kb_used_data":217328,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93681344,"statfs":{"total":96636764160,"available":95929696256,"internally_reserved":0,"allocated":222543872,"data_stored":222330811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710416,"kb_used_data":215548,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":93661424,"statfs":{"total":96636764160,"available":95909298176,"internally_reserved":0,"allocated":220721152,"data_stored":220508091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611616,"kb_used_data":214800,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93760224,"statfs":{"total":96636764160,"available":96010469376,"internally_reserved":0,"allocated":219955200,"data_stored":219742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704900,"kb_used_data":213740,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93666940,"statfs":{"total":96636764160,"available":95914946560,"internally_reserved":0,"allocated":218869760,"data_stored":218656699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502918863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217694208,"data_stored":217694208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219332608,"data_stored":219332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219860992,"data_stored":219860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221581312,"data_stored":221581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:42.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:42.984 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:42.985 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:43.012 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-10T20:26:42.035869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":440235016,"num_objects":53744,"num_object_clones":0,"num_object_copies":214976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120601,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2598,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140727,"ondisk_log_size":140727,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717428,"kb_used_data":861416,"kb_used_omap":14,"kb_used_meta":1855921,"kb_avail":374769932,"statfs":{"total":386547056640,"available":383764410368,"internally_reserved":0,"allocated":882089984,"data_stored":881237740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1900463670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,2,6,18,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20766720,"num_objects":-2535,"num_object_clones":0,"num_object_copies":-10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2535,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004421"},"pg_stats":[{"pgid":"1.f","version":"40'8730","reported_seq":9012,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.215889+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:37.215889+0000","last_peered":"2024-09-10T20:26:37.215889+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:37.215889+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27787264,"num_objects":3392,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8899","reported_seq":9031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:39.936224+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:39.936224+0000","last_peered":"2024-09-10T20:26:39.936224+0000","last_clean":"2024-09-10T20:26:39.936224+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:39.936224+0000","last_undegraded":"2024-09-10T20:26:39.936224+0000","last_fullsized":"2024-09-10T20:26:39.936224+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8899,"log_dups_size":0,"ondisk_log_size":8899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27467776,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8774","reported_seq":8977,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.075624+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:39.075624+0000","last_peered":"2024-09-10T20:26:39.075624+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:39.075624+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8774,"log_dups_size":0,"ondisk_log_size":8774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27572840,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8773,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8992","reported_seq":13161,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:40.073587+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:40.073587+0000","last_peered":"2024-09-10T20:26:40.073587+0000","last_clean":"2024-09-10T20:26:40.073587+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:40.073587+0000","last_undegraded":"2024-09-10T20:26:40.073587+0000","last_fullsized":"2024-09-10T20:26:40.073587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27967488,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8817","reported_seq":9135,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.943643+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:36.943643+0000","last_peered":"2024-09-10T20:26:36.943643+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:36.943643+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456672,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8742","reported_seq":8964,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.150551+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:37.150551+0000","last_peered":"2024-09-10T20:26:37.150551+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:37.150551+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27492352,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8786","reported_seq":8989,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.529887+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:41.529887+0000","last_peered":"2024-09-10T20:26:41.529887+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:41.529887+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26899616,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8930","reported_seq":9098,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.262817+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:41.262817+0000","last_peered":"2024-09-10T20:26:41.262817+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:41.262817+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27546784,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8667","reported_seq":12639,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:36.858158+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:36.858158+0000","last_peered":"2024-09-10T20:26:36.858158+0000","last_clean":"2024-09-10T20:26:36.858158+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:36.858158+0000","last_undegraded":"2024-09-10T20:26:36.858158+0000","last_fullsized":"2024-09-10T20:26:36.858158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8667,"log_dups_size":0,"ondisk_log_size":8667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27310672,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8716","reported_seq":12686,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:39.173016+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:39.173016+0000","last_peered":"2024-09-10T20:26:39.173016+0000","last_clean":"2024-09-10T20:26:39.173016+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:39.173016+0000","last_undegraded":"2024-09-10T20:26:39.173016+0000","last_fullsized":"2024-09-10T20:26:39.173016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28065792,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8716","reported_seq":8989,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:36.981626+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:36.981626+0000","last_peered":"2024-09-10T20:26:36.981626+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:36.981626+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27823856,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8806","reported_seq":9104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.214619+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:37.214619+0000","last_peered":"2024-09-10T20:26:37.214619+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:37.214619+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27515472,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8805,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8597","reported_seq":8780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.122713+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:39.122713+0000","last_peered":"2024-09-10T20:26:39.122713+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:39.122713+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491472,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1331}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8744","reported_seq":9013,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:37.118775+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:37.118775+0000","last_peered":"2024-09-10T20:26:37.118775+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:37.118775+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8744,"log_dups_size":0,"ondisk_log_size":8744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440288,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1475}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8975","reported_seq":9180,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.066458+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:39.066458+0000","last_peered":"2024-09-10T20:26:39.066458+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:39.066458+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8836","reported_seq":12297,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.542561+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:41.542561+0000","last_peered":"2024-09-10T20:26:41.542561+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:41.542561+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26940576,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":440235016,"num_objects":53744,"num_object_clones":0,"num_object_copies":214976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120601,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2598,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":878469120,"data_stored":878469120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140727,"ondisk_log_size":140727,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690496,"kb_used_data":217328,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93681344,"statfs":{"total":96636764160,"available":95929696256,"internally_reserved":0,"allocated":222543872,"data_stored":222330811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710416,"kb_used_data":215548,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":93661424,"statfs":{"total":96636764160,"available":95909298176,"internally_reserved":0,"allocated":220721152,"data_stored":220508091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611616,"kb_used_data":214800,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93760224,"statfs":{"total":96636764160,"available":96010469376,"internally_reserved":0,"allocated":219955200,"data_stored":219742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704900,"kb_used_data":213740,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93666940,"statfs":{"total":96636764160,"available":95914946560,"internally_reserved":0,"allocated":218869760,"data_stored":218656699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502918863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217694208,"data_stored":217694208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219332608,"data_stored":219332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219860992,"data_stored":219860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221581312,"data_stored":221581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:43.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:43.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8730", 'reported_seq': 9012, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:37.215889+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:37.215889+0000', 'last_peered': '2024-09-10T20:26:37.215889+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:37.215889+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8730, 'log_dups_size': 0, 'ondisk_log_size': 8730, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27787264, 'num_objects': 3392, 'num_object_clones': 0, 'num_object_copies': 13568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8730, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1537}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8774", 'reported_seq': 8977, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:39.075624+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:39.075624+0000', 'last_peered': '2024-09-10T20:26:39.075624+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:39.075624+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8774, 'log_dups_size': 0, 'ondisk_log_size': 8774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27572840, 'num_objects': 3367, 'num_object_clones': 0, 'num_object_copies': 13468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3367, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8773, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1459}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8817", 'reported_seq': 9135, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:36.943643+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:36.943643+0000', 'last_peered': '2024-09-10T20:26:36.943643+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:36.943643+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8817, 'log_dups_size': 0, 'ondisk_log_size': 8817, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27456672, 'num_objects': 3352, 'num_object_clones': 0, 'num_object_copies': 13408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8815, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8742", 'reported_seq': 8964, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:37.150551+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:37.150551+0000', 'last_peered': '2024-09-10T20:26:37.150551+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:37.150551+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8742, 'log_dups_size': 0, 'ondisk_log_size': 8742, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27492352, 'num_objects': 3356, 'num_object_clones': 0, 'num_object_copies': 13424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8742, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1499}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8786", 'reported_seq': 8989, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:41.529887+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:41.529887+0000', 'last_peered': '2024-09-10T20:26:41.529887+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:41.529887+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8786, 'log_dups_size': 0, 'ondisk_log_size': 8786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26899616, 'num_objects': 3284, 'num_object_clones': 0, 'num_object_copies': 13136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8784, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1391}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:43.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8930", 'reported_seq': 9098, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:41.262817+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:41.262817+0000', 'last_peered': '2024-09-10T20:26:41.262817+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:41.262817+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8930, 'log_dups_size': 0, 'ondisk_log_size': 8930, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27546784, 'num_objects': 3363, 'num_object_clones': 0, 'num_object_copies': 13452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8928, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8716", 'reported_seq': 8989, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:36.981626+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:36.981626+0000', 'last_peered': '2024-09-10T20:26:36.981626+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:36.981626+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8716, 'log_dups_size': 0, 'ondisk_log_size': 8716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27823856, 'num_objects': 3397, 'num_object_clones': 0, 'num_object_copies': 13588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8713, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1541}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8806", 'reported_seq': 9104, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:37.214619+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:37.214619+0000', 'last_peered': '2024-09-10T20:26:37.214619+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:37.214619+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8806, 'log_dups_size': 0, 'ondisk_log_size': 8806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27515472, 'num_objects': 3359, 'num_object_clones': 0, 'num_object_copies': 13436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8805, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1499}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8597", 'reported_seq': 8780, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:39.122713+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:39.122713+0000', 'last_peered': '2024-09-10T20:26:39.122713+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:39.122713+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8597, 'log_dups_size': 0, 'ondisk_log_size': 8597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26491472, 'num_objects': 3234, 'num_object_clones': 0, 'num_object_copies': 12936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8596, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8744", 'reported_seq': 9013, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:37.118775+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:37.118775+0000', 'last_peered': '2024-09-10T20:26:37.118775+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:37.118775+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8744, 'log_dups_size': 0, 'ondisk_log_size': 8744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27440288, 'num_objects': 3350, 'num_object_clones': 0, 'num_object_copies': 13400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8742, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1475}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:43.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'8975", 'reported_seq': 9180, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:39.066458+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:39.066458+0000', 'last_peered': '2024-09-10T20:26:39.066458+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:39.066458+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8975, 'log_dups_size': 0, 'ondisk_log_size': 8975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28456096, 'num_objects': 3474, 'num_object_clones': 0, 'num_object_copies': 13896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8973, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1566}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:43.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:43.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8836", 'reported_seq': 12297, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:41.542561+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:41.542561+0000', 'last_peered': '2024-09-10T20:26:41.542561+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:41.542561+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8836, 'log_dups_size': 0, 'ondisk_log_size': 8836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26940576, 'num_objects': 3289, 'num_object_clones': 0, 'num_object_copies': 13156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8834, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1691, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2997}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:43.016 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:43.784 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:43.784 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21078/214976 objects degraded (9.805%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":492,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":53744,"data_bytes":440235016,"bytes_used":2782646272,"bytes_avail":383764410368,"bytes_total":386547056640,"degraded_objects":21078,"degraded_total":214976,"degraded_ratio":0.098048154212563257,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":259,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":211},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:43.817 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:43.817 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:44.435 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:44.477 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-10T20:26:44.036918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":436401160,"num_objects":53276,"num_object_clones":0,"num_object_copies":213104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121069,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2598,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141195,"ondisk_log_size":141195,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731080,"kb_used_data":857008,"kb_used_omap":14,"kb_used_meta":1873969,"kb_avail":374756280,"statfs":{"total":386547056640,"available":383750430720,"internally_reserved":0,"allocated":877576192,"data_stored":876723948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1918944822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,4,10,12,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22716416,"num_objects":-2773,"num_object_clones":0,"num_object_copies":-11092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2773,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004790"},"pg_stats":[{"pgid":"1.f","version":"40'8800","reported_seq":9081,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:42.085958+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:42.085958+0000","last_peered":"2024-09-10T20:26:42.085958+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:42.085958+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27213824,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8899","reported_seq":9031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:39.936224+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:39.936224+0000","last_peered":"2024-09-10T20:26:39.936224+0000","last_clean":"2024-09-10T20:26:39.936224+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:39.936224+0000","last_undegraded":"2024-09-10T20:26:39.936224+0000","last_fullsized":"2024-09-10T20:26:39.936224+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8899,"log_dups_size":0,"ondisk_log_size":8899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27467776,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8774","reported_seq":8977,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.075624+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:39.075624+0000","last_peered":"2024-09-10T20:26:39.075624+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:39.075624+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8774,"log_dups_size":0,"ondisk_log_size":8774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27572840,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8773,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'8992","reported_seq":13161,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:40.073587+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:40.073587+0000","last_peered":"2024-09-10T20:26:40.073587+0000","last_clean":"2024-09-10T20:26:40.073587+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:40.073587+0000","last_undegraded":"2024-09-10T20:26:40.073587+0000","last_fullsized":"2024-09-10T20:26:40.073587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27967488,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8874","reported_seq":9194,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:42.093657+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:42.093657+0000","last_peered":"2024-09-10T20:26:42.093657+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:42.093657+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26989728,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8820","reported_seq":9042,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:42.092377+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:42.092377+0000","last_peered":"2024-09-10T20:26:42.092377+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:42.092377+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8820,"log_dups_size":0,"ondisk_log_size":8820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26853376,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8820,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8786","reported_seq":8989,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.529887+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:41.529887+0000","last_peered":"2024-09-10T20:26:41.529887+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:41.529887+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26899616,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8930","reported_seq":9098,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.262817+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:41.262817+0000","last_peered":"2024-09-10T20:26:41.262817+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:41.262817+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27546784,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8730","reported_seq":12702,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:41.990844+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:41.990844+0000","last_peered":"2024-09-10T20:26:41.990844+0000","last_clean":"2024-09-10T20:26:41.990844+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:41.990844+0000","last_undegraded":"2024-09-10T20:26:41.990844+0000","last_fullsized":"2024-09-10T20:26:41.990844+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26794576,"num_objects":3271,"num_object_clones":0,"num_object_copies":13084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8716","reported_seq":12686,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:39.173016+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:39.173016+0000","last_peered":"2024-09-10T20:26:39.173016+0000","last_clean":"2024-09-10T20:26:39.173016+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:39.173016+0000","last_undegraded":"2024-09-10T20:26:39.173016+0000","last_fullsized":"2024-09-10T20:26:39.173016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28065792,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8773","reported_seq":9046,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:42.211308+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:42.211308+0000","last_peered":"2024-09-10T20:26:42.211308+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:42.211308+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27356912,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8864","reported_seq":9162,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:42.068837+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:42.068837+0000","last_peered":"2024-09-10T20:26:42.068837+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:42.068837+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27040336,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1441}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8597","reported_seq":8780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.122713+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:39.122713+0000","last_peered":"2024-09-10T20:26:39.122713+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:39.122713+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491472,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1331}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8829","reported_seq":9098,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:42.182210+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:42.182210+0000","last_peered":"2024-09-10T20:26:42.182210+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:42.182210+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743968,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1390}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'8975","reported_seq":9180,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:39.066458+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:39.066458+0000","last_peered":"2024-09-10T20:26:39.066458+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:39.066458+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8836","reported_seq":12297,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:41.542561+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:41.542561+0000","last_peered":"2024-09-10T20:26:41.542561+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:41.542561+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26940576,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":436401160,"num_objects":53276,"num_object_clones":0,"num_object_copies":213104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121069,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2598,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":874033152,"data_stored":874033152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141195,"ondisk_log_size":141195,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117083,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704148,"kb_used_data":212920,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93667692,"statfs":{"total":96636764160,"available":95915716608,"internally_reserved":0,"allocated":218030080,"data_stored":217817019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":502984400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710416,"kb_used_data":215548,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":93661424,"statfs":{"total":96636764160,"available":95909298176,"internally_reserved":0,"allocated":220721152,"data_stored":220508091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611616,"kb_used_data":214800,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93760224,"statfs":{"total":96636764160,"available":96010469376,"internally_reserved":0,"allocated":219955200,"data_stored":219742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704900,"kb_used_data":213740,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93666940,"statfs":{"total":96636764160,"available":95914946560,"internally_reserved":0,"allocated":218869760,"data_stored":218656699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502918863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217694208,"data_stored":217694208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219332608,"data_stored":219332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219860992,"data_stored":219860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217145344,"data_stored":217145344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:47.479 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:48.262 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:48.262 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:48.284 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-10T20:26:48.038541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":426882056,"num_objects":52114,"num_object_clones":0,"num_object_copies":208456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122231,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2703,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142357,"ondisk_log_size":142357,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680252,"kb_used_data":839444,"kb_used_omap":14,"kb_used_meta":1840689,"kb_avail":374807108,"statfs":{"total":386547056640,"available":383802478592,"internally_reserved":0,"allocated":859590656,"data_stored":858738412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1884866102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,7,1,10,15,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24354816,"num_objects":-2973,"num_object_clones":0,"num_object_copies":-11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004930"},"pg_stats":[{"pgid":"1.f","version":"40'8890","reported_seq":9172,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.338283+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:47.338283+0000","last_peered":"2024-09-10T20:26:47.338283+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:47.338283+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":3232,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8962","reported_seq":9094,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:45.114737+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:45.114737+0000","last_peered":"2024-09-10T20:26:45.114737+0000","last_clean":"2024-09-10T20:26:45.114737+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:45.114737+0000","last_undegraded":"2024-09-10T20:26:45.114737+0000","last_fullsized":"2024-09-10T20:26:45.114737+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26951680,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8844","reported_seq":9047,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:44.260941+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:44.260941+0000","last_peered":"2024-09-10T20:26:44.260941+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:44.260941+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8844,"log_dups_size":0,"ondisk_log_size":8844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26999400,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8843,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9055","reported_seq":13224,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:45.097420+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:45.097420+0000","last_peered":"2024-09-10T20:26:45.097420+0000","last_clean":"2024-09-10T20:26:45.097420+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:45.097420+0000","last_undegraded":"2024-09-10T20:26:45.097420+0000","last_fullsized":"2024-09-10T20:26:45.097420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27451392,"num_objects":3351,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8949","reported_seq":9269,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.298770+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:47.298770+0000","last_peered":"2024-09-10T20:26:47.298770+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:47.298770+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375328,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8947,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8901","reported_seq":9123,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.113538+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:47.113538+0000","last_peered":"2024-09-10T20:26:47.113538+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:47.113538+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26189824,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8854","reported_seq":9056,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.570800+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:46.570800+0000","last_peered":"2024-09-10T20:26:46.570800+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:46.570800+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8992","reported_seq":9160,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.425811+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:46.425811+0000","last_peered":"2024-09-10T20:26:46.425811+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:46.425811+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038880,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8806","reported_seq":12778,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:47.278550+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:47.278550+0000","last_peered":"2024-09-10T20:26:47.278550+0000","last_clean":"2024-09-10T20:26:47.278550+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:47.278550+0000","last_undegraded":"2024-09-10T20:26:47.278550+0000","last_fullsized":"2024-09-10T20:26:47.278550+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26171984,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5359,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8789","reported_seq":12758,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:44.184065+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:44.184065+0000","last_peered":"2024-09-10T20:26:44.184065+0000","last_clean":"2024-09-10T20:26:44.184065+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:44.184065+0000","last_undegraded":"2024-09-10T20:26:44.184065+0000","last_fullsized":"2024-09-10T20:26:44.184065+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8789,"log_dups_size":0,"ondisk_log_size":8789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27467776,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8857","reported_seq":9129,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.258572+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:47.258572+0000","last_peered":"2024-09-10T20:26:47.258572+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:47.258572+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26668784,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8945","reported_seq":9243,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.214964+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:47.214964+0000","last_peered":"2024-09-10T20:26:47.214964+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:47.214964+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8945,"log_dups_size":0,"ondisk_log_size":8945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26376784,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8944,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8661","reported_seq":8844,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:44.195390+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:44.195390+0000","last_peered":"2024-09-10T20:26:44.195390+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:44.195390+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8661,"log_dups_size":0,"ondisk_log_size":8661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25967184,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8896","reported_seq":9165,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.240290+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:47.240290+0000","last_peered":"2024-09-10T20:26:47.240290+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:47.240290+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1323}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9052","reported_seq":9257,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:44.054144+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:44.054144+0000","last_peered":"2024-09-10T20:26:44.054144+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:44.054144+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27825312,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8904","reported_seq":12575,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.545412+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:46.545412+0000","last_peered":"2024-09-10T20:26:46.545412+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:46.545412+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8904,"log_dups_size":0,"ondisk_log_size":8904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26383520,"num_objects":3221,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8902,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3034}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":426882056,"num_objects":52114,"num_object_clones":0,"num_object_copies":208456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122231,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2703,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":855453696,"data_stored":855453696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142357,"ondisk_log_size":142357,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687288,"kb_used_data":208552,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93684552,"statfs":{"total":96636764160,"available":95932981248,"internally_reserved":0,"allocated":213557248,"data_stored":213344187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":490204880},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693628,"kb_used_data":211244,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93678212,"statfs":{"total":96636764160,"available":95926489088,"internally_reserved":0,"allocated":216313856,"data_stored":216100795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,0,2,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611452,"kb_used_data":210436,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93760388,"statfs":{"total":96636764160,"available":96010637312,"internally_reserved":0,"allocated":215486464,"data_stored":215273403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":410578642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687884,"kb_used_data":209212,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93683956,"statfs":{"total":96636764160,"available":95932370944,"internally_reserved":0,"allocated":214233088,"data_stored":214020027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":490139343},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,3,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213086208,"data_stored":213086208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":214700032,"data_stored":214700032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215379968,"data_stored":215379968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212287488,"data_stored":212287488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:48.285 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:48.845 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:48.873 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-10T20:26:48.038541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":426882056,"num_objects":52114,"num_object_clones":0,"num_object_copies":208456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122231,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2703,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142357,"ondisk_log_size":142357,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680252,"kb_used_data":839444,"kb_used_omap":14,"kb_used_meta":1840689,"kb_avail":374807108,"statfs":{"total":386547056640,"available":383802478592,"internally_reserved":0,"allocated":859590656,"data_stored":858738412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1884866102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,7,1,10,15,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24354816,"num_objects":-2973,"num_object_clones":0,"num_object_copies":-11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004930"},"pg_stats":[{"pgid":"1.f","version":"40'8890","reported_seq":9172,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.338283+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:47.338283+0000","last_peered":"2024-09-10T20:26:47.338283+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:47.338283+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":3232,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8962","reported_seq":9094,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:45.114737+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:45.114737+0000","last_peered":"2024-09-10T20:26:45.114737+0000","last_clean":"2024-09-10T20:26:45.114737+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:45.114737+0000","last_undegraded":"2024-09-10T20:26:45.114737+0000","last_fullsized":"2024-09-10T20:26:45.114737+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26951680,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8844","reported_seq":9047,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:44.260941+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:44.260941+0000","last_peered":"2024-09-10T20:26:44.260941+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:44.260941+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8844,"log_dups_size":0,"ondisk_log_size":8844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26999400,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8843,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9055","reported_seq":13224,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:45.097420+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:45.097420+0000","last_peered":"2024-09-10T20:26:45.097420+0000","last_clean":"2024-09-10T20:26:45.097420+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:45.097420+0000","last_undegraded":"2024-09-10T20:26:45.097420+0000","last_fullsized":"2024-09-10T20:26:45.097420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27451392,"num_objects":3351,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8949","reported_seq":9269,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.298770+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:47.298770+0000","last_peered":"2024-09-10T20:26:47.298770+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:47.298770+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375328,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8947,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8901","reported_seq":9123,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.113538+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:47.113538+0000","last_peered":"2024-09-10T20:26:47.113538+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:47.113538+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26189824,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8854","reported_seq":9056,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.570800+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:46.570800+0000","last_peered":"2024-09-10T20:26:46.570800+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:46.570800+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8992","reported_seq":9160,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.425811+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:46.425811+0000","last_peered":"2024-09-10T20:26:46.425811+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:46.425811+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038880,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8806","reported_seq":12778,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:47.278550+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:47.278550+0000","last_peered":"2024-09-10T20:26:47.278550+0000","last_clean":"2024-09-10T20:26:47.278550+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:47.278550+0000","last_undegraded":"2024-09-10T20:26:47.278550+0000","last_fullsized":"2024-09-10T20:26:47.278550+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26171984,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5359,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8789","reported_seq":12758,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:44.184065+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:44.184065+0000","last_peered":"2024-09-10T20:26:44.184065+0000","last_clean":"2024-09-10T20:26:44.184065+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:44.184065+0000","last_undegraded":"2024-09-10T20:26:44.184065+0000","last_fullsized":"2024-09-10T20:26:44.184065+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8789,"log_dups_size":0,"ondisk_log_size":8789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27467776,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8857","reported_seq":9129,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.258572+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:47.258572+0000","last_peered":"2024-09-10T20:26:47.258572+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:47.258572+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26668784,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8945","reported_seq":9243,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.214964+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:47.214964+0000","last_peered":"2024-09-10T20:26:47.214964+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:47.214964+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8945,"log_dups_size":0,"ondisk_log_size":8945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26376784,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8944,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8661","reported_seq":8844,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:44.195390+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:44.195390+0000","last_peered":"2024-09-10T20:26:44.195390+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:44.195390+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8661,"log_dups_size":0,"ondisk_log_size":8661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25967184,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8896","reported_seq":9165,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.240290+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:47.240290+0000","last_peered":"2024-09-10T20:26:47.240290+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:47.240290+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1323}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9052","reported_seq":9257,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:44.054144+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:44.054144+0000","last_peered":"2024-09-10T20:26:44.054144+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:44.054144+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27825312,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8904","reported_seq":12575,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.545412+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:46.545412+0000","last_peered":"2024-09-10T20:26:46.545412+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:46.545412+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8904,"log_dups_size":0,"ondisk_log_size":8904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26383520,"num_objects":3221,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8902,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3034}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":426882056,"num_objects":52114,"num_object_clones":0,"num_object_copies":208456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122231,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2703,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":855453696,"data_stored":855453696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142357,"ondisk_log_size":142357,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687288,"kb_used_data":208552,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93684552,"statfs":{"total":96636764160,"available":95932981248,"internally_reserved":0,"allocated":213557248,"data_stored":213344187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":490204880},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693628,"kb_used_data":211244,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93678212,"statfs":{"total":96636764160,"available":95926489088,"internally_reserved":0,"allocated":216313856,"data_stored":216100795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,0,2,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611452,"kb_used_data":210436,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93760388,"statfs":{"total":96636764160,"available":96010637312,"internally_reserved":0,"allocated":215486464,"data_stored":215273403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":410578642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687884,"kb_used_data":209212,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93683956,"statfs":{"total":96636764160,"available":95932370944,"internally_reserved":0,"allocated":214233088,"data_stored":214020027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":490139343},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,3,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213086208,"data_stored":213086208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":214700032,"data_stored":214700032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215379968,"data_stored":215379968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212287488,"data_stored":212287488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8890", 'reported_seq': 9172, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:47.338283+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:47.338283+0000', 'last_peered': '2024-09-10T20:26:47.338283+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:47.338283+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8890, 'log_dups_size': 0, 'ondisk_log_size': 8890, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26476544, 'num_objects': 3232, 'num_object_clones': 0, 'num_object_copies': 12928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8890, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1377}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8844", 'reported_seq': 9047, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:44.260941+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:44.260941+0000', 'last_peered': '2024-09-10T20:26:44.260941+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:44.260941+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8844, 'log_dups_size': 0, 'ondisk_log_size': 8844, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26999400, 'num_objects': 3297, 'num_object_clones': 0, 'num_object_copies': 13188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3297, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8843, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1389}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'8949", 'reported_seq': 9269, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:47.298770+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:47.298770+0000', 'last_peered': '2024-09-10T20:26:47.298770+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:47.298770+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8949, 'log_dups_size': 0, 'ondisk_log_size': 8949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26375328, 'num_objects': 3220, 'num_object_clones': 0, 'num_object_copies': 12880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8947, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1365}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8901", 'reported_seq': 9123, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:47.113538+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:47.113538+0000', 'last_peered': '2024-09-10T20:26:47.113538+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:47.113538+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8901, 'log_dups_size': 0, 'ondisk_log_size': 8901, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26189824, 'num_objects': 3197, 'num_object_clones': 0, 'num_object_copies': 12788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8901, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:48.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8854", 'reported_seq': 9056, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:46.570800+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:46.570800+0000', 'last_peered': '2024-09-10T20:26:46.570800+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:46.570800+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8854, 'log_dups_size': 0, 'ondisk_log_size': 8854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26342560, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8852, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1323}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'8992", 'reported_seq': 9160, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:46.425811+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:46.425811+0000', 'last_peered': '2024-09-10T20:26:46.425811+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:46.425811+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8992, 'log_dups_size': 0, 'ondisk_log_size': 8992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27038880, 'num_objects': 3301, 'num_object_clones': 0, 'num_object_copies': 13204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8990, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8857", 'reported_seq': 9129, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:47.258572+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:47.258572+0000', 'last_peered': '2024-09-10T20:26:47.258572+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:47.258572+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8857, 'log_dups_size': 0, 'ondisk_log_size': 8857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26668784, 'num_objects': 3256, 'num_object_clones': 0, 'num_object_copies': 13024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8854, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1400}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'8945", 'reported_seq': 9243, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:47.214964+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:47.214964+0000', 'last_peered': '2024-09-10T20:26:47.214964+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:47.214964+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8945, 'log_dups_size': 0, 'ondisk_log_size': 8945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26376784, 'num_objects': 3220, 'num_object_clones': 0, 'num_object_copies': 12880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8944, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1360}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:48.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8661", 'reported_seq': 8844, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:44.195390+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:44.195390+0000', 'last_peered': '2024-09-10T20:26:44.195390+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:44.195390+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8661, 'log_dups_size': 0, 'ondisk_log_size': 8661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25967184, 'num_objects': 3170, 'num_object_clones': 0, 'num_object_copies': 12680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8660, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1267}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:48.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:48.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8896", 'reported_seq': 9165, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:47.240290+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:47.240290+0000', 'last_peered': '2024-09-10T20:26:47.240290+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:47.240290+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8896, 'log_dups_size': 0, 'ondisk_log_size': 8896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26195104, 'num_objects': 3198, 'num_object_clones': 0, 'num_object_copies': 12792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8894, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:48.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:48.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'9052", 'reported_seq': 9257, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:44.054144+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:44.054144+0000', 'last_peered': '2024-09-10T20:26:44.054144+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:44.054144+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9052, 'log_dups_size': 0, 'ondisk_log_size': 9052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27825312, 'num_objects': 3397, 'num_object_clones': 0, 'num_object_copies': 13588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9050, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:48.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:48.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8904", 'reported_seq': 12575, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:46.545412+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:46.545412+0000', 'last_peered': '2024-09-10T20:26:46.545412+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:46.545412+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8904, 'log_dups_size': 0, 'ondisk_log_size': 8904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26383520, 'num_objects': 3221, 'num_object_clones': 0, 'num_object_copies': 12884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 187, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8902, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1796, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3034}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:48.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:49.659 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20973/208456 objects degraded (10.061%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":498,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52114,"data_bytes":426882056,"bytes_used":2744578048,"bytes_avail":383802478592,"bytes_total":386547056640,"degraded_objects":20973,"degraded_total":208456,"degraded_ratio":0.10061116014890432,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":302,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":247},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:49.679 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:49.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:50.311 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:50.328 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-10T20:26:50.039334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424801288,"num_objects":51860,"num_object_clones":0,"num_object_copies":207440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122485,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2703,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142611,"ondisk_log_size":142611,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2679732,"kb_used_data":834748,"kb_used_omap":14,"kb_used_meta":1844849,"kb_avail":374807628,"statfs":{"total":386547056640,"available":383803011072,"internally_reserved":0,"allocated":854781952,"data_stored":853929708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1889125942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,8,1,14,13,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20578304,"num_objects":-2512,"num_object_clones":0,"num_object_copies":-10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004899"},"pg_stats":[{"pgid":"1.f","version":"40'8890","reported_seq":9172,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.338283+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:47.338283+0000","last_peered":"2024-09-10T20:26:47.338283+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:47.338283+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":3232,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'8962","reported_seq":9094,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:45.114737+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:45.114737+0000","last_peered":"2024-09-10T20:26:45.114737+0000","last_clean":"2024-09-10T20:26:45.114737+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:45.114737+0000","last_undegraded":"2024-09-10T20:26:45.114737+0000","last_fullsized":"2024-09-10T20:26:45.114737+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26951680,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8902","reported_seq":9105,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.218622+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:49.218622+0000","last_peered":"2024-09-10T20:26:49.218622+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:49.218622+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8902,"log_dups_size":0,"ondisk_log_size":8902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26524264,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8901,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9055","reported_seq":13224,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:45.097420+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:45.097420+0000","last_peered":"2024-09-10T20:26:45.097420+0000","last_clean":"2024-09-10T20:26:45.097420+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:45.097420+0000","last_undegraded":"2024-09-10T20:26:45.097420+0000","last_fullsized":"2024-09-10T20:26:45.097420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27451392,"num_objects":3351,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8949","reported_seq":9269,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.298770+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:47.298770+0000","last_peered":"2024-09-10T20:26:47.298770+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:47.298770+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375328,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8947,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8901","reported_seq":9123,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.113538+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:47.113538+0000","last_peered":"2024-09-10T20:26:47.113538+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:47.113538+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26189824,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8854","reported_seq":9056,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.570800+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:46.570800+0000","last_peered":"2024-09-10T20:26:46.570800+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:46.570800+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'8992","reported_seq":9160,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.425811+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:46.425811+0000","last_peered":"2024-09-10T20:26:46.425811+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:46.425811+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038880,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8806","reported_seq":12778,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:47.278550+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:47.278550+0000","last_peered":"2024-09-10T20:26:47.278550+0000","last_clean":"2024-09-10T20:26:47.278550+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:47.278550+0000","last_undegraded":"2024-09-10T20:26:47.278550+0000","last_fullsized":"2024-09-10T20:26:47.278550+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26171984,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5359,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8856","reported_seq":12826,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:49.041313+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:49.041313+0000","last_peered":"2024-09-10T20:26:49.041313+0000","last_clean":"2024-09-10T20:26:49.041313+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:49.041313+0000","last_undegraded":"2024-09-10T20:26:49.041313+0000","last_fullsized":"2024-09-10T20:26:49.041313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26918912,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8857","reported_seq":9129,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.258572+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:47.258572+0000","last_peered":"2024-09-10T20:26:47.258572+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:47.258572+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26668784,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8945","reported_seq":9243,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.214964+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:47.214964+0000","last_peered":"2024-09-10T20:26:47.214964+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:47.214964+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8945,"log_dups_size":0,"ondisk_log_size":8945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26376784,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8944,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8714","reported_seq":8897,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.237397+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:49.237397+0000","last_peered":"2024-09-10T20:26:49.237397+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:49.237397+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8714,"log_dups_size":0,"ondisk_log_size":8714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25533008,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8896","reported_seq":9165,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.240290+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:47.240290+0000","last_peered":"2024-09-10T20:26:47.240290+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:47.240290+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1323}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9128","reported_seq":9333,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.164467+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:49.164467+0000","last_peered":"2024-09-10T20:26:49.164467+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:49.164467+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202720,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8904","reported_seq":12575,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:46.545412+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:46.545412+0000","last_peered":"2024-09-10T20:26:46.545412+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:46.545412+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8904,"log_dups_size":0,"ondisk_log_size":8904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26383520,"num_objects":3221,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8902,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3034}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":424801288,"num_objects":51860,"num_object_clones":0,"num_object_copies":207440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122485,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2703,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":850624512,"data_stored":850624512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142611,"ondisk_log_size":142611,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687288,"kb_used_data":208552,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93684552,"statfs":{"total":96636764160,"available":95932981248,"internally_reserved":0,"allocated":213557248,"data_stored":213344187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":490204880},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693108,"kb_used_data":206548,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93678732,"statfs":{"total":96636764160,"available":95927021568,"internally_reserved":0,"allocated":211505152,"data_stored":211292091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,6,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611452,"kb_used_data":210436,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93760388,"statfs":{"total":96636764160,"available":96010637312,"internally_reserved":0,"allocated":215486464,"data_stored":215273403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":410578642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687884,"kb_used_data":209212,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93683956,"statfs":{"total":96636764160,"available":95932370944,"internally_reserved":0,"allocated":214233088,"data_stored":214020027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":490139343},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,3,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213086208,"data_stored":213086208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":214700032,"data_stored":214700032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210550784,"data_stored":210550784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212287488,"data_stored":212287488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:53.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:53.902 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:53.902 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:53.927 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-10T20:26:52.040164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":421827592,"num_objects":51497,"num_object_clones":0,"num_object_copies":205988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122848,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142974,"ondisk_log_size":142974,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2662332,"kb_used_data":825696,"kb_used_omap":14,"kb_used_meta":1836529,"kb_avail":374825028,"statfs":{"total":386547056640,"available":383820828672,"internally_reserved":0,"allocated":845512704,"data_stored":844660460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1880606262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,1,17,12,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21323776,"num_objects":-2603,"num_object_clones":0,"num_object_copies":-10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2603,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004906"},"pg_stats":[{"pgid":"1.f","version":"40'8890","reported_seq":9172,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.338283+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:47.338283+0000","last_peered":"2024-09-10T20:26:47.338283+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:47.338283+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":3232,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'9040","reported_seq":9172,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:49.998572+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:49.998572+0000","last_peered":"2024-09-10T20:26:49.998572+0000","last_clean":"2024-09-10T20:26:49.998572+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:49.998572+0000","last_undegraded":"2024-09-10T20:26:49.998572+0000","last_fullsized":"2024-09-10T20:26:49.998572+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26312704,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8902","reported_seq":9105,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.218622+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:49.218622+0000","last_peered":"2024-09-10T20:26:49.218622+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:49.218622+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8902,"log_dups_size":0,"ondisk_log_size":8902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26524264,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8901,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9126","reported_seq":13295,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:50.183518+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:50.183518+0000","last_peered":"2024-09-10T20:26:50.183518+0000","last_clean":"2024-09-10T20:26:50.183518+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:50.183518+0000","last_undegraded":"2024-09-10T20:26:50.183518+0000","last_fullsized":"2024-09-10T20:26:50.183518+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3386,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'8949","reported_seq":9269,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.298770+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:47.298770+0000","last_peered":"2024-09-10T20:26:47.298770+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:47.298770+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375328,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8947,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8901","reported_seq":9123,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.113538+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:47.113538+0000","last_peered":"2024-09-10T20:26:47.113538+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:47.113538+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26189824,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8927","reported_seq":9130,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.557850+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:51.557850+0000","last_peered":"2024-09-10T20:26:51.557850+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:51.557850+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8927,"log_dups_size":0,"ondisk_log_size":8927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744544,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8925,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1250}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'9063","reported_seq":9231,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.653671+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:51.653671+0000","last_peered":"2024-09-10T20:26:51.653671+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:51.653671+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9063,"log_dups_size":0,"ondisk_log_size":9063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26457248,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8806","reported_seq":12778,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:47.278550+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:47.278550+0000","last_peered":"2024-09-10T20:26:47.278550+0000","last_clean":"2024-09-10T20:26:47.278550+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:47.278550+0000","last_undegraded":"2024-09-10T20:26:47.278550+0000","last_fullsized":"2024-09-10T20:26:47.278550+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26171984,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5359,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8856","reported_seq":12826,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:49.041313+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:49.041313+0000","last_peered":"2024-09-10T20:26:49.041313+0000","last_clean":"2024-09-10T20:26:49.041313+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:49.041313+0000","last_undegraded":"2024-09-10T20:26:49.041313+0000","last_fullsized":"2024-09-10T20:26:49.041313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26918912,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8857","reported_seq":9129,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.258572+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:47.258572+0000","last_peered":"2024-09-10T20:26:47.258572+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:47.258572+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26668784,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'8945","reported_seq":9243,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.214964+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:47.214964+0000","last_peered":"2024-09-10T20:26:47.214964+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:47.214964+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8945,"log_dups_size":0,"ondisk_log_size":8945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26376784,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8944,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8714","reported_seq":8897,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.237397+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:49.237397+0000","last_peered":"2024-09-10T20:26:49.237397+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:49.237397+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8714,"log_dups_size":0,"ondisk_log_size":8714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25533008,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8896","reported_seq":9165,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:47.240290+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:47.240290+0000","last_peered":"2024-09-10T20:26:47.240290+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:47.240290+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1323}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9128","reported_seq":9333,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.164467+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:49.164467+0000","last_peered":"2024-09-10T20:26:49.164467+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:49.164467+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202720,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8974","reported_seq":12774,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.687561+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:51.687561+0000","last_peered":"2024-09-10T20:26:51.687561+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:51.687561+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8974,"log_dups_size":0,"ondisk_log_size":8974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25810080,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8972,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":421827592,"num_objects":51497,"num_object_clones":0,"num_object_copies":205988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122848,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":841293824,"data_stored":841293824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142974,"ondisk_log_size":142974,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687288,"kb_used_data":208552,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93684552,"statfs":{"total":96636764160,"available":95932981248,"internally_reserved":0,"allocated":213557248,"data_stored":213344187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":490204880},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693108,"kb_used_data":206548,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93678732,"statfs":{"total":96636764160,"available":95927021568,"internally_reserved":0,"allocated":211505152,"data_stored":211292091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,6,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594260,"kb_used_data":205764,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93777580,"statfs":{"total":96636764160,"available":96028241920,"internally_reserved":0,"allocated":210702336,"data_stored":210489275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687676,"kb_used_data":204832,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93684164,"statfs":{"total":96636764160,"available":95932583936,"internally_reserved":0,"allocated":209747968,"data_stored":209534907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494399183},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208416768,"data_stored":208416768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210038784,"data_stored":210038784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210550784,"data_stored":210550784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212287488,"data_stored":212287488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:53.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:54.530 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:54.531 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:54.557 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-10T20:26:54.041008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417543176,"num_objects":50974,"num_object_clones":0,"num_object_copies":203896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123371,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143497,"ondisk_log_size":143497,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680004,"kb_used_data":821156,"kb_used_omap":14,"kb_used_meta":1858737,"kb_avail":374807356,"statfs":{"total":386547056640,"available":383802732544,"internally_reserved":0,"allocated":840863744,"data_stored":840011500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1903347254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,3,15,16,21,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22691840,"num_objects":-2770,"num_object_clones":0,"num_object_copies":-11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005139"},"pg_stats":[{"pgid":"1.f","version":"40'8958","reported_seq":9240,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.387179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:52.387179+0000","last_peered":"2024-09-10T20:26:52.387179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:52.387179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8958,"log_dups_size":0,"ondisk_log_size":8958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25919488,"num_objects":3164,"num_object_clones":0,"num_object_copies":12656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'9040","reported_seq":9172,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:49.998572+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:49.998572+0000","last_peered":"2024-09-10T20:26:49.998572+0000","last_clean":"2024-09-10T20:26:49.998572+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:49.998572+0000","last_undegraded":"2024-09-10T20:26:49.998572+0000","last_fullsized":"2024-09-10T20:26:49.998572+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26312704,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8902","reported_seq":9105,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.218622+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:49.218622+0000","last_peered":"2024-09-10T20:26:49.218622+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:49.218622+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8902,"log_dups_size":0,"ondisk_log_size":8902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26524264,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8901,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9126","reported_seq":13295,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:50.183518+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:50.183518+0000","last_peered":"2024-09-10T20:26:50.183518+0000","last_clean":"2024-09-10T20:26:50.183518+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:50.183518+0000","last_undegraded":"2024-09-10T20:26:50.183518+0000","last_fullsized":"2024-09-10T20:26:50.183518+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3386,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9031","reported_seq":9351,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.426815+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:52.426815+0000","last_peered":"2024-09-10T20:26:52.426815+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:52.426815+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703584,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8992","reported_seq":9214,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.320389+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:52.320389+0000","last_peered":"2024-09-10T20:26:52.320389+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:52.320389+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25444352,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8992,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1249}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8927","reported_seq":9130,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.557850+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:51.557850+0000","last_peered":"2024-09-10T20:26:51.557850+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:51.557850+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8927,"log_dups_size":0,"ondisk_log_size":8927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744544,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8925,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1250}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'9063","reported_seq":9231,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.653671+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:51.653671+0000","last_peered":"2024-09-10T20:26:51.653671+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:51.653671+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9063,"log_dups_size":0,"ondisk_log_size":9063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26457248,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8876","reported_seq":12848,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:52.309518+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:52.309518+0000","last_peered":"2024-09-10T20:26:52.309518+0000","last_clean":"2024-09-10T20:26:52.309518+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:52.309518+0000","last_undegraded":"2024-09-10T20:26:52.309518+0000","last_fullsized":"2024-09-10T20:26:52.309518+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8876,"log_dups_size":0,"ondisk_log_size":8876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25598544,"num_objects":3125,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5429,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8856","reported_seq":12826,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:49.041313+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:49.041313+0000","last_peered":"2024-09-10T20:26:49.041313+0000","last_clean":"2024-09-10T20:26:49.041313+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:49.041313+0000","last_undegraded":"2024-09-10T20:26:49.041313+0000","last_fullsized":"2024-09-10T20:26:49.041313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26918912,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8923","reported_seq":9196,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.387015+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:52.387015+0000","last_peered":"2024-09-10T20:26:52.387015+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:52.387015+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26128112,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9019","reported_seq":9317,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.386046+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:52.386046+0000","last_peered":"2024-09-10T20:26:52.386046+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:52.386046+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25770576,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1286}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8714","reported_seq":8897,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.237397+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:49.237397+0000","last_peered":"2024-09-10T20:26:49.237397+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:49.237397+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8714,"log_dups_size":0,"ondisk_log_size":8714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25533008,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8968","reported_seq":9237,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.385512+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:52.385512+0000","last_peered":"2024-09-10T20:26:52.385512+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:52.385512+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8968,"log_dups_size":0,"ondisk_log_size":8968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605280,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1251}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9128","reported_seq":9333,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.164467+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:49.164467+0000","last_peered":"2024-09-10T20:26:49.164467+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:49.164467+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202720,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8974","reported_seq":12774,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.687561+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:51.687561+0000","last_peered":"2024-09-10T20:26:51.687561+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:51.687561+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8974,"log_dups_size":0,"ondisk_log_size":8974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25810080,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8972,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417543176,"num_objects":50974,"num_object_clones":0,"num_object_copies":203896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123371,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":836620288,"data_stored":836620288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143497,"ondisk_log_size":143497,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117085,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704960,"kb_used_data":204012,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93666880,"statfs":{"total":96636764160,"available":95914885120,"internally_reserved":0,"allocated":208908288,"data_stored":208695227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,9,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693108,"kb_used_data":206548,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93678732,"statfs":{"total":96636764160,"available":95927021568,"internally_reserved":0,"allocated":211505152,"data_stored":211292091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,6,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594260,"kb_used_data":205764,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93777580,"statfs":{"total":96636764160,"available":96028241920,"internally_reserved":0,"allocated":210702336,"data_stored":210489275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687676,"kb_used_data":204832,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93684164,"statfs":{"total":96636764160,"available":95932583936,"internally_reserved":0,"allocated":209747968,"data_stored":209534907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494399183},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208416768,"data_stored":208416768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210038784,"data_stored":210038784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210550784,"data_stored":210550784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207613952,"data_stored":207613952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:54.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'8958", 'reported_seq': 9240, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:52.387179+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:52.387179+0000', 'last_peered': '2024-09-10T20:26:52.387179+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:52.387179+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8958, 'log_dups_size': 0, 'ondisk_log_size': 8958, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25919488, 'num_objects': 3164, 'num_object_clones': 0, 'num_object_copies': 12656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8958, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'8902", 'reported_seq': 9105, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:49.218622+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:49.218622+0000', 'last_peered': '2024-09-10T20:26:49.218622+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:49.218622+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8902, 'log_dups_size': 0, 'ondisk_log_size': 8902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26524264, 'num_objects': 3239, 'num_object_clones': 0, 'num_object_copies': 12956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3239, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8901, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1331}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'9031", 'reported_seq': 9351, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:52.426815+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:52.426815+0000', 'last_peered': '2024-09-10T20:26:52.426815+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:52.426815+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9031, 'log_dups_size': 0, 'ondisk_log_size': 9031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25703584, 'num_objects': 3138, 'num_object_clones': 0, 'num_object_copies': 12552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9029, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1283}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'8992", 'reported_seq': 9214, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:52.320389+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:52.320389+0000', 'last_peered': '2024-09-10T20:26:52.320389+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:52.320389+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8992, 'log_dups_size': 0, 'ondisk_log_size': 8992, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25444352, 'num_objects': 3106, 'num_object_clones': 0, 'num_object_copies': 12424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8992, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1249}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'8927", 'reported_seq': 9130, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:51.557850+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:51.557850+0000', 'last_peered': '2024-09-10T20:26:51.557850+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:51.557850+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8927, 'log_dups_size': 0, 'ondisk_log_size': 8927, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25744544, 'num_objects': 3143, 'num_object_clones': 0, 'num_object_copies': 12572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8925, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1250}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:26:54.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'9063", 'reported_seq': 9231, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:51.653671+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:51.653671+0000', 'last_peered': '2024-09-10T20:26:51.653671+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:51.653671+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9063, 'log_dups_size': 0, 'ondisk_log_size': 9063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26457248, 'num_objects': 3230, 'num_object_clones': 0, 'num_object_copies': 12920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9061, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8923", 'reported_seq': 9196, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:52.387015+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:52.387015+0000', 'last_peered': '2024-09-10T20:26:52.387015+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:52.387015+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8923, 'log_dups_size': 0, 'ondisk_log_size': 8923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26128112, 'num_objects': 3190, 'num_object_clones': 0, 'num_object_copies': 12760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8920, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1334}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'9019", 'reported_seq': 9317, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:52.386046+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:52.386046+0000', 'last_peered': '2024-09-10T20:26:52.386046+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:52.386046+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9019, 'log_dups_size': 0, 'ondisk_log_size': 9019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25770576, 'num_objects': 3146, 'num_object_clones': 0, 'num_object_copies': 12584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9018, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8714", 'reported_seq': 8897, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:49.237397+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:49.237397+0000', 'last_peered': '2024-09-10T20:26:49.237397+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:49.237397+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8714, 'log_dups_size': 0, 'ondisk_log_size': 8714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25533008, 'num_objects': 3117, 'num_object_clones': 0, 'num_object_copies': 12468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8713, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1214}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'8968", 'reported_seq': 9237, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:52.385512+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:52.385512+0000', 'last_peered': '2024-09-10T20:26:52.385512+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:52.385512+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8968, 'log_dups_size': 0, 'ondisk_log_size': 8968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25605280, 'num_objects': 3126, 'num_object_clones': 0, 'num_object_copies': 12504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8966, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1251}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'9128", 'reported_seq': 9333, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:49.164467+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:49.164467+0000', 'last_peered': '2024-09-10T20:26:49.164467+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:49.164467+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9128, 'log_dups_size': 0, 'ondisk_log_size': 9128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27202720, 'num_objects': 3321, 'num_object_clones': 0, 'num_object_copies': 13284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9126, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:26:54.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:26:54.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'8974", 'reported_seq': 12774, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:51.687561+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:51.687561+0000', 'last_peered': '2024-09-10T20:26:51.687561+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:51.687561+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8974, 'log_dups_size': 0, 'ondisk_log_size': 8974, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25810080, 'num_objects': 3151, 'num_object_clones': 0, 'num_object_copies': 12604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 122, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8972, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1861, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:26:54.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:26:55.356 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:55.356 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20908/203896 objects degraded (10.254%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50974,"data_bytes":417543176,"bytes_used":2744324096,"bytes_avail":383802732544,"bytes_total":386547056640,"degraded_objects":20908,"degraded_total":203896,"degraded_ratio":0.10254247263310708,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":230},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:26:55.385 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:26:55.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:56.018 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:56.018 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:56.040 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-10T20:26:54.041008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417543176,"num_objects":50974,"num_object_clones":0,"num_object_copies":203896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123371,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143497,"ondisk_log_size":143497,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680004,"kb_used_data":821156,"kb_used_omap":14,"kb_used_meta":1858737,"kb_avail":374807356,"statfs":{"total":386547056640,"available":383802732544,"internally_reserved":0,"allocated":840863744,"data_stored":840011500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1903347254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,3,15,16,21,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22691840,"num_objects":-2770,"num_object_clones":0,"num_object_copies":-11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005139"},"pg_stats":[{"pgid":"1.f","version":"40'8958","reported_seq":9240,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.387179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:52.387179+0000","last_peered":"2024-09-10T20:26:52.387179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:52.387179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8958,"log_dups_size":0,"ondisk_log_size":8958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25919488,"num_objects":3164,"num_object_clones":0,"num_object_copies":12656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'9040","reported_seq":9172,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:49.998572+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:49.998572+0000","last_peered":"2024-09-10T20:26:49.998572+0000","last_clean":"2024-09-10T20:26:49.998572+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:49.998572+0000","last_undegraded":"2024-09-10T20:26:49.998572+0000","last_fullsized":"2024-09-10T20:26:49.998572+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26312704,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8902","reported_seq":9105,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.218622+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:49.218622+0000","last_peered":"2024-09-10T20:26:49.218622+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:49.218622+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8902,"log_dups_size":0,"ondisk_log_size":8902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26524264,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8901,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9126","reported_seq":13295,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:50.183518+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:50.183518+0000","last_peered":"2024-09-10T20:26:50.183518+0000","last_clean":"2024-09-10T20:26:50.183518+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:50.183518+0000","last_undegraded":"2024-09-10T20:26:50.183518+0000","last_fullsized":"2024-09-10T20:26:50.183518+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3386,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9031","reported_seq":9351,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.426815+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:52.426815+0000","last_peered":"2024-09-10T20:26:52.426815+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:52.426815+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703584,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'8992","reported_seq":9214,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.320389+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:52.320389+0000","last_peered":"2024-09-10T20:26:52.320389+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:52.320389+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25444352,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8992,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1249}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'8927","reported_seq":9130,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.557850+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:51.557850+0000","last_peered":"2024-09-10T20:26:51.557850+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:51.557850+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8927,"log_dups_size":0,"ondisk_log_size":8927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744544,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8925,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1250}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'9063","reported_seq":9231,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.653671+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:51.653671+0000","last_peered":"2024-09-10T20:26:51.653671+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:51.653671+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9063,"log_dups_size":0,"ondisk_log_size":9063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26457248,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8876","reported_seq":12848,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:52.309518+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:52.309518+0000","last_peered":"2024-09-10T20:26:52.309518+0000","last_clean":"2024-09-10T20:26:52.309518+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:52.309518+0000","last_undegraded":"2024-09-10T20:26:52.309518+0000","last_fullsized":"2024-09-10T20:26:52.309518+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8876,"log_dups_size":0,"ondisk_log_size":8876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25598544,"num_objects":3125,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5429,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8856","reported_seq":12826,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:49.041313+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:49.041313+0000","last_peered":"2024-09-10T20:26:49.041313+0000","last_clean":"2024-09-10T20:26:49.041313+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:49.041313+0000","last_undegraded":"2024-09-10T20:26:49.041313+0000","last_fullsized":"2024-09-10T20:26:49.041313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26918912,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8923","reported_seq":9196,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.387015+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:52.387015+0000","last_peered":"2024-09-10T20:26:52.387015+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:52.387015+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26128112,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9019","reported_seq":9317,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.386046+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:52.386046+0000","last_peered":"2024-09-10T20:26:52.386046+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:52.386046+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25770576,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1286}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8714","reported_seq":8897,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.237397+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:49.237397+0000","last_peered":"2024-09-10T20:26:49.237397+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:49.237397+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8714,"log_dups_size":0,"ondisk_log_size":8714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25533008,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'8968","reported_seq":9237,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:52.385512+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:52.385512+0000","last_peered":"2024-09-10T20:26:52.385512+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:52.385512+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8968,"log_dups_size":0,"ondisk_log_size":8968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605280,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1251}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9128","reported_seq":9333,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:49.164467+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:49.164467+0000","last_peered":"2024-09-10T20:26:49.164467+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:49.164467+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202720,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'8974","reported_seq":12774,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:51.687561+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:51.687561+0000","last_peered":"2024-09-10T20:26:51.687561+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:51.687561+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8974,"log_dups_size":0,"ondisk_log_size":8974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25810080,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8972,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417543176,"num_objects":50974,"num_object_clones":0,"num_object_copies":203896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123371,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":836620288,"data_stored":836620288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143497,"ondisk_log_size":143497,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117085,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704960,"kb_used_data":204012,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93666880,"statfs":{"total":96636764160,"available":95914885120,"internally_reserved":0,"allocated":208908288,"data_stored":208695227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,9,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693108,"kb_used_data":206548,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93678732,"statfs":{"total":96636764160,"available":95927021568,"internally_reserved":0,"allocated":211505152,"data_stored":211292091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,6,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594260,"kb_used_data":205764,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93777580,"statfs":{"total":96636764160,"available":96028241920,"internally_reserved":0,"allocated":210702336,"data_stored":210489275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687676,"kb_used_data":204832,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93684164,"statfs":{"total":96636764160,"available":95932583936,"internally_reserved":0,"allocated":209747968,"data_stored":209534907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494399183},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208416768,"data_stored":208416768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210038784,"data_stored":210038784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210550784,"data_stored":210550784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207613952,"data_stored":207613952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:59.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:59.698 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:26:59.699 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:26:59.726 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-10T20:26:59.487215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":408040456,"num_objects":49814,"num_object_clones":0,"num_object_copies":199256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124531,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2815,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144657,"ondisk_log_size":144657,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735480,"kb_used_data":801652,"kb_used_omap":14,"kb_used_meta":1933681,"kb_avail":374751880,"statfs":{"total":386547056640,"available":383745925120,"internally_reserved":0,"allocated":820891648,"data_stored":820039404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1980089910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,6,4,8,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18841600,"num_objects":-2300,"num_object_clones":0,"num_object_copies":-9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.448674"},"pg_stats":[{"pgid":"1.f","version":"40'9024","reported_seq":9306,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.229771+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:57.229771+0000","last_peered":"2024-09-10T20:26:57.229771+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:57.229771+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25378816,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9024,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'9111","reported_seq":9243,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:55.261615+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:55.261615+0000","last_peered":"2024-09-10T20:26:55.261615+0000","last_clean":"2024-09-10T20:26:55.261615+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:55.261615+0000","last_undegraded":"2024-09-10T20:26:55.261615+0000","last_fullsized":"2024-09-10T20:26:55.261615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9111,"log_dups_size":0,"ondisk_log_size":9111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25731072,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'8980","reported_seq":9183,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:54.373856+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:54.373856+0000","last_peered":"2024-09-10T20:26:54.373856+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:54.373856+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25885288,"num_objects":3161,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8979,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9190","reported_seq":13359,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:55.181648+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:55.181648+0000","last_peered":"2024-09-10T20:26:55.181648+0000","last_clean":"2024-09-10T20:26:55.181648+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:55.181648+0000","last_undegraded":"2024-09-10T20:26:55.181648+0000","last_fullsized":"2024-09-10T20:26:55.181648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3450,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9094","reported_seq":9414,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.353633+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:57.353633+0000","last_peered":"2024-09-10T20:26:57.353633+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:57.353633+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25187488,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'9058","reported_seq":9280,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.424167+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:57.424167+0000","last_peered":"2024-09-10T20:26:57.424167+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:57.424167+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903680,"num_objects":3040,"num_object_clones":0,"num_object_copies":12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9058,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'9014","reported_seq":9217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.507776+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:56.507776+0000","last_peered":"2024-09-10T20:26:56.507776+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:56.507776+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25031840,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9012,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'9157","reported_seq":9322,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.557613+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:56.557613+0000","last_peered":"2024-09-10T20:26:56.557613+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:56.557613+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9157,"log_dups_size":0,"ondisk_log_size":9157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25687200,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9155,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8949","reported_seq":12921,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:57.269394+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:57.269394+0000","last_peered":"2024-09-10T20:26:57.269394+0000","last_clean":"2024-09-10T20:26:57.269394+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:57.269394+0000","last_undegraded":"2024-09-10T20:26:57.269394+0000","last_fullsized":"2024-09-10T20:26:57.269394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25000528,"num_objects":3052,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5502,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8923","reported_seq":12893,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:54.154580+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:54.154580+0000","last_peered":"2024-09-10T20:26:54.154580+0000","last_clean":"2024-09-10T20:26:54.154580+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:54.154580+0000","last_undegraded":"2024-09-10T20:26:54.154580+0000","last_fullsized":"2024-09-10T20:26:54.154580+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26370048,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8994","reported_seq":9265,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.463603+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:57.463603+0000","last_peered":"2024-09-10T20:26:57.463603+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:57.463603+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25546480,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9090","reported_seq":9388,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.350798+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:57.350798+0000","last_peered":"2024-09-10T20:26:57.350798+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:57.350798+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25188944,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1215}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8776","reported_seq":8959,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:54.254552+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:54.254552+0000","last_peered":"2024-09-10T20:26:54.254552+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:54.254552+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8776,"log_dups_size":0,"ondisk_log_size":8776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25025104,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8775,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1152}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'9035","reported_seq":9304,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.397871+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:57.397871+0000","last_peered":"2024-09-10T20:26:57.397871+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:57.397871+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9035,"log_dups_size":0,"ondisk_log_size":9035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9033,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9216","reported_seq":9421,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:54.428039+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:54.428039+0000","last_peered":"2024-09-10T20:26:54.428039+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:54.428039+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26481824,"num_objects":3233,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'9046","reported_seq":12939,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.673222+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:56.673222+0000","last_peered":"2024-09-10T20:26:56.673222+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:56.673222+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9046,"log_dups_size":0,"ondisk_log_size":9046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25220256,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1908,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":408040456,"num_objects":49814,"num_object_clones":0,"num_object_copies":199256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124531,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2815,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":817606656,"data_stored":817606656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144657,"ondisk_log_size":144657,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707804,"kb_used_data":198516,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93664036,"statfs":{"total":96636764160,"available":95911972864,"internally_reserved":0,"allocated":203280384,"data_stored":203067323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,0,1,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710684,"kb_used_data":201924,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93661156,"statfs":{"total":96636764160,"available":95909023744,"internally_reserved":0,"allocated":206770176,"data_stored":206557115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611772,"kb_used_data":201056,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93760068,"statfs":{"total":96636764160,"available":96010309632,"internally_reserved":0,"allocated":205881344,"data_stored":205668283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705220,"kb_used_data":200156,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93666620,"statfs":{"total":96636764160,"available":95914618880,"internally_reserved":0,"allocated":204959744,"data_stored":204746683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203710464,"data_stored":203710464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205168640,"data_stored":205168640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":205783040,"data_stored":205783040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202944512,"data_stored":202944512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:59.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:00.301 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:00.302 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:00.323 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-10T20:27:00.043034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405861384,"num_objects":49548,"num_object_clones":0,"num_object_copies":198192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49548,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124797,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2815,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144923,"ondisk_log_size":144923,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718676,"kb_used_data":797304,"kb_used_omap":14,"kb_used_meta":1921201,"kb_avail":374768684,"statfs":{"total":386547056640,"available":383763132416,"internally_reserved":0,"allocated":816439296,"data_stored":815587052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1967310390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,0,9,6,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18939904,"num_objects":-2312,"num_object_clones":0,"num_object_copies":-9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003700"},"pg_stats":[{"pgid":"1.f","version":"40'9024","reported_seq":9306,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.229771+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:57.229771+0000","last_peered":"2024-09-10T20:26:57.229771+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:57.229771+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25378816,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9024,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'9111","reported_seq":9243,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:55.261615+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:26:55.261615+0000","last_peered":"2024-09-10T20:26:55.261615+0000","last_clean":"2024-09-10T20:26:55.261615+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:26:55.261615+0000","last_undegraded":"2024-09-10T20:26:55.261615+0000","last_fullsized":"2024-09-10T20:26:55.261615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9111,"log_dups_size":0,"ondisk_log_size":9111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25731072,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'9055","reported_seq":9258,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.345268+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:59.345268+0000","last_peered":"2024-09-10T20:26:59.345268+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:59.345268+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25270888,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9054,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9190","reported_seq":13359,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:55.181648+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:26:55.181648+0000","last_peered":"2024-09-10T20:26:55.181648+0000","last_clean":"2024-09-10T20:26:55.181648+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:26:55.181648+0000","last_undegraded":"2024-09-10T20:26:55.181648+0000","last_fullsized":"2024-09-10T20:26:55.181648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3450,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9094","reported_seq":9414,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.353633+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:57.353633+0000","last_peered":"2024-09-10T20:26:57.353633+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:57.353633+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25187488,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'9058","reported_seq":9280,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.424167+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:57.424167+0000","last_peered":"2024-09-10T20:26:57.424167+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:57.424167+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903680,"num_objects":3040,"num_object_clones":0,"num_object_copies":12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9058,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'9014","reported_seq":9217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.507776+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:56.507776+0000","last_peered":"2024-09-10T20:26:56.507776+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:56.507776+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25031840,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9012,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'9157","reported_seq":9322,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.557613+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:56.557613+0000","last_peered":"2024-09-10T20:26:56.557613+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:56.557613+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9157,"log_dups_size":0,"ondisk_log_size":9157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25687200,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9155,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8949","reported_seq":12921,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:57.269394+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:57.269394+0000","last_peered":"2024-09-10T20:26:57.269394+0000","last_clean":"2024-09-10T20:26:57.269394+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:57.269394+0000","last_undegraded":"2024-09-10T20:26:57.269394+0000","last_fullsized":"2024-09-10T20:26:57.269394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25000528,"num_objects":3052,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5502,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8991","reported_seq":12961,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:59.390238+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:59.390238+0000","last_peered":"2024-09-10T20:26:59.390238+0000","last_clean":"2024-09-10T20:26:59.390238+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:59.390238+0000","last_undegraded":"2024-09-10T20:26:59.390238+0000","last_fullsized":"2024-09-10T20:26:59.390238+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25812992,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8994","reported_seq":9265,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.463603+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:57.463603+0000","last_peered":"2024-09-10T20:26:57.463603+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:57.463603+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25546480,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9090","reported_seq":9388,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.350798+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:57.350798+0000","last_peered":"2024-09-10T20:26:57.350798+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:57.350798+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25188944,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1215}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8839","reported_seq":9022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.286332+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:59.286332+0000","last_peered":"2024-09-10T20:26:59.286332+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:59.286332+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24509008,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'9035","reported_seq":9304,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.397871+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:57.397871+0000","last_peered":"2024-09-10T20:26:57.397871+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:57.397871+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9035,"log_dups_size":0,"ondisk_log_size":9035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9033,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9276","reported_seq":9480,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.412775+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:59.412775+0000","last_peered":"2024-09-10T20:26:59.412775+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:59.412775+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990304,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'9046","reported_seq":12939,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.673222+0000","last_change":"2024-09-10T20:25:00.528460+0000","last_active":"2024-09-10T20:26:56.673222+0000","last_peered":"2024-09-10T20:26:56.673222+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:56.673222+0000","last_undegraded":"2024-09-10T20:20:15.494071+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9046,"log_dups_size":0,"ondisk_log_size":9046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25220256,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1908,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405861384,"num_objects":49548,"num_object_clones":0,"num_object_copies":198192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49548,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124797,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2815,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":812990464,"data_stored":812990464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144923,"ondisk_log_size":144923,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707804,"kb_used_data":198516,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93664036,"statfs":{"total":96636764160,"available":95911972864,"internally_reserved":0,"allocated":203280384,"data_stored":203067323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,0,1,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693880,"kb_used_data":197576,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93677960,"statfs":{"total":96636764160,"available":95926231040,"internally_reserved":0,"allocated":202317824,"data_stored":202104763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611772,"kb_used_data":201056,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93760068,"statfs":{"total":96636764160,"available":96010309632,"internally_reserved":0,"allocated":205881344,"data_stored":205668283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705220,"kb_used_data":200156,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93666620,"statfs":{"total":96636764160,"available":95914618880,"internally_reserved":0,"allocated":204959744,"data_stored":204746683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203710464,"data_stored":203710464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205168640,"data_stored":205168640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201166848,"data_stored":201166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202944512,"data_stored":202944512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:00.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:00.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'9024", 'reported_seq': 9306, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:57.229771+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:26:57.229771+0000', 'last_peered': '2024-09-10T20:26:57.229771+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:26:57.229771+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9024, 'log_dups_size': 0, 'ondisk_log_size': 9024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25378816, 'num_objects': 3098, 'num_object_clones': 0, 'num_object_copies': 12392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9024, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1243}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:00.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:00.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'9055", 'reported_seq': 9258, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:59.345268+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:59.345268+0000', 'last_peered': '2024-09-10T20:26:59.345268+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:59.345268+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9055, 'log_dups_size': 0, 'ondisk_log_size': 9055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25270888, 'num_objects': 3086, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3086, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9054, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1178}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:00.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:00.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'9094", 'reported_seq': 9414, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:57.353633+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:26:57.353633+0000', 'last_peered': '2024-09-10T20:26:57.353633+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:26:57.353633+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9094, 'log_dups_size': 0, 'ondisk_log_size': 9094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25187488, 'num_objects': 3075, 'num_object_clones': 0, 'num_object_copies': 12300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9092, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1220}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:00.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:00.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'9058", 'reported_seq': 9280, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:57.424167+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:26:57.424167+0000', 'last_peered': '2024-09-10T20:26:57.424167+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:26:57.424167+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9058, 'log_dups_size': 0, 'ondisk_log_size': 9058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24903680, 'num_objects': 3040, 'num_object_clones': 0, 'num_object_copies': 12160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9058, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "40'9014", 'reported_seq': 9217, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:56.507776+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:26:56.507776+0000', 'last_peered': '2024-09-10T20:26:56.507776+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:26:56.507776+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9014, 'log_dups_size': 0, 'ondisk_log_size': 9014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25031840, 'num_objects': 3056, 'num_object_clones': 0, 'num_object_copies': 12224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9012, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1163}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'9157", 'reported_seq': 9322, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:56.557613+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:26:56.557613+0000', 'last_peered': '2024-09-10T20:26:56.557613+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:26:56.557613+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9157, 'log_dups_size': 0, 'ondisk_log_size': 9157, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25687200, 'num_objects': 3136, 'num_object_clones': 0, 'num_object_copies': 12544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9155, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1136}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'8994", 'reported_seq': 9265, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:57.463603+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:26:57.463603+0000', 'last_peered': '2024-09-10T20:26:57.463603+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:26:57.463603+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8994, 'log_dups_size': 0, 'ondisk_log_size': 8994, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25546480, 'num_objects': 3119, 'num_object_clones': 0, 'num_object_copies': 12476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8991, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'9090", 'reported_seq': 9388, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:57.350798+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:26:57.350798+0000', 'last_peered': '2024-09-10T20:26:57.350798+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:26:57.350798+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9090, 'log_dups_size': 0, 'ondisk_log_size': 9090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25188944, 'num_objects': 3075, 'num_object_clones': 0, 'num_object_copies': 12300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9089, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1215}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:00.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8839", 'reported_seq': 9022, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:59.286332+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:59.286332+0000', 'last_peered': '2024-09-10T20:26:59.286332+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:59.286332+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8839, 'log_dups_size': 0, 'ondisk_log_size': 8839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24509008, 'num_objects': 2992, 'num_object_clones': 0, 'num_object_copies': 11968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8838, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:00.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:00.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'9035", 'reported_seq': 9304, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:57.397871+0000', 'last_change': '2024-09-10T20:20:16.099421+0000', 'last_active': '2024-09-10T20:26:57.397871+0000', 'last_peered': '2024-09-10T20:26:57.397871+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:26:57.397871+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9035, 'log_dups_size': 0, 'ondisk_log_size': 9035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25056416, 'num_objects': 3059, 'num_object_clones': 0, 'num_object_copies': 12236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1877, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9033, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:00.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:00.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'9276", 'reported_seq': 9480, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:59.412775+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:59.412775+0000', 'last_peered': '2024-09-10T20:26:59.412775+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:59.412775+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9276, 'log_dups_size': 0, 'ondisk_log_size': 9276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25990304, 'num_objects': 3173, 'num_object_clones': 0, 'num_object_copies': 12692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9274, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:00.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:27:00.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'9046", 'reported_seq': 12939, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:56.673222+0000', 'last_change': '2024-09-10T20:25:00.528460+0000', 'last_active': '2024-09-10T20:26:56.673222+0000', 'last_peered': '2024-09-10T20:26:56.673222+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:56.673222+0000', 'last_undegraded': '2024-09-10T20:20:15.494071+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9046, 'log_dups_size': 0, 'ondisk_log_size': 9046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25220256, 'num_objects': 3079, 'num_object_clones': 0, 'num_object_copies': 12316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 75, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9044, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1908, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:00.327 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:01.050 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:01.051 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20861/198192 objects degraded (10.526%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":509,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49548,"data_bytes":405861384,"bytes_used":2783924224,"bytes_avail":383763132416,"bytes_total":386547056640,"degraded_objects":20861,"degraded_total":198192,"degraded_ratio":0.10525651893113748,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":231},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:27:01.081 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:01.082 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:01.216 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:01.206+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9103 (0'0,42'9103] local-lis/les=41/42 n=3022 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9103 lcod 42'9102 mlcod 42'9102 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:01.503 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:01.498+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9105 (0'0,42'9105] local-lis/les=41/42 n=3020 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9105 lcod 42'9104 mlcod 42'9104 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:01.675 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:01.671+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9107 (0'0,42'9107] local-lis/les=41/42 n=3018 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9105 crt=42'9105 lcod 42'9104 mlcod 42'9104 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:01.737 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:01.763 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-10T20:27:00.524372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404362248,"num_objects":49365,"num_object_clones":0,"num_object_copies":197460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124980,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2890,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145106,"ondisk_log_size":145106,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722376,"kb_used_data":792688,"kb_used_omap":14,"kb_used_meta":1929521,"kb_avail":374764984,"statfs":{"total":386547056640,"available":383759343616,"internally_reserved":0,"allocated":811712512,"data_stored":810855551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1975830070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,7,8,11,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17465344,"num_objects":-2132,"num_object_clones":0,"num_object_copies":-8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.484208"},"pg_stats":[{"pgid":"1.f","version":"40'9024","reported_seq":9306,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.229771+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:26:57.229771+0000","last_peered":"2024-09-10T20:26:57.229771+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:26:57.229771+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25378816,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9024,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"41'9174","reported_seq":9308,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-10T20:27:00.179142+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:00.179142+0000","last_peered":"2024-09-10T20:27:00.179142+0000","last_clean":"2024-09-10T20:27:00.179142+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:00.179142+0000","last_undegraded":"2024-09-10T20:27:00.179142+0000","last_fullsized":"2024-09-10T20:27:00.179142+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25214976,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'9055","reported_seq":9258,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.345268+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:59.345268+0000","last_peered":"2024-09-10T20:26:59.345268+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:59.345268+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25270888,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9054,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"41'9271","reported_seq":13442,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-10T20:27:00.333650+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:00.333650+0000","last_peered":"2024-09-10T20:27:00.333650+0000","last_clean":"2024-09-10T20:27:00.333650+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:00.333650+0000","last_undegraded":"2024-09-10T20:27:00.333650+0000","last_fullsized":"2024-09-10T20:27:00.333650+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25681920,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9094","reported_seq":9414,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.353633+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:26:57.353633+0000","last_peered":"2024-09-10T20:26:57.353633+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:26:57.353633+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25187488,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'9058","reported_seq":9280,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.424167+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:26:57.424167+0000","last_peered":"2024-09-10T20:26:57.424167+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:26:57.424167+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903680,"num_objects":3040,"num_object_clones":0,"num_object_copies":12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9058,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'9014","reported_seq":9217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.507776+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:26:56.507776+0000","last_peered":"2024-09-10T20:26:56.507776+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:26:56.507776+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25031840,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9012,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'9157","reported_seq":9322,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:56.557613+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:26:56.557613+0000","last_peered":"2024-09-10T20:26:56.557613+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:26:56.557613+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9157,"log_dups_size":0,"ondisk_log_size":9157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25687200,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9155,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'8949","reported_seq":12921,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:57.269394+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:26:57.269394+0000","last_peered":"2024-09-10T20:26:57.269394+0000","last_clean":"2024-09-10T20:26:57.269394+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:26:57.269394+0000","last_undegraded":"2024-09-10T20:26:57.269394+0000","last_fullsized":"2024-09-10T20:26:57.269394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25000528,"num_objects":3052,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5502,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8991","reported_seq":12961,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:59.390238+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:59.390238+0000","last_peered":"2024-09-10T20:26:59.390238+0000","last_clean":"2024-09-10T20:26:59.390238+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:59.390238+0000","last_undegraded":"2024-09-10T20:26:59.390238+0000","last_fullsized":"2024-09-10T20:26:59.390238+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25812992,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'8994","reported_seq":9265,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.463603+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:26:57.463603+0000","last_peered":"2024-09-10T20:26:57.463603+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:26:57.463603+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25546480,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9090","reported_seq":9388,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.350798+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:26:57.350798+0000","last_peered":"2024-09-10T20:26:57.350798+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:26:57.350798+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25188944,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1215}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8839","reported_seq":9022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.286332+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:59.286332+0000","last_peered":"2024-09-10T20:26:59.286332+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:59.286332+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24509008,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'9035","reported_seq":9304,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:57.397871+0000","last_change":"2024-09-10T20:20:16.099421+0000","last_active":"2024-09-10T20:26:57.397871+0000","last_peered":"2024-09-10T20:26:57.397871+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:26:57.397871+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9035,"log_dups_size":0,"ondisk_log_size":9035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9033,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9276","reported_seq":9480,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.412775+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:59.412775+0000","last_peered":"2024-09-10T20:26:59.412775+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:59.412775+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990304,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'9085","reported_seq":13135,"reported_epoch":41,"state":"peering","last_fresh":"2024-09-10T20:26:59.492624+0000","last_change":"2024-09-10T20:27:00.423930+0000","last_active":"2024-09-10T20:27:00.423930+0000","last_peered":"2024-09-10T20:26:59.492624+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:59.492624+0000","last_undegraded":"2024-09-10T20:26:59.492624+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24900768,"num_objects":3040,"num_object_clones":0,"num_object_copies":12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9083,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1983,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":404362248,"num_objects":49365,"num_object_clones":0,"num_object_copies":197460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124980,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2890,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":808476672,"data_stored":808476672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145106,"ondisk_log_size":145106,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707804,"kb_used_data":198516,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93664036,"statfs":{"total":96636764160,"available":95911972864,"internally_reserved":0,"allocated":203280384,"data_stored":203067323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,0,1,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693880,"kb_used_data":197576,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93677960,"statfs":{"total":96636764160,"available":95926231040,"internally_reserved":0,"allocated":202317824,"data_stored":202104763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615472,"kb_used_data":196440,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93756368,"statfs":{"total":96636764160,"available":96006520832,"internally_reserved":0,"allocated":201154560,"data_stored":200936782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705220,"kb_used_data":200156,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93666620,"statfs":{"total":96636764160,"available":95914618880,"internally_reserved":0,"allocated":204959744,"data_stored":204746683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203710464,"data_stored":203710464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200654848,"data_stored":200654848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201166848,"data_stored":201166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202944512,"data_stored":202944512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:01.765 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:01.761+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9108 (0'0,42'9108] local-lis/les=41/42 n=3017 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9107 crt=42'9106 lcod 42'9106 mlcod 42'9106 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:01.966 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:01.962+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9110 (0'0,42'9110] local-lis/les=41/42 n=3015 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9110 lcod 42'9109 mlcod 42'9109 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:02.067 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:02.062+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9111 (0'0,42'9111] local-lis/les=41/42 n=3014 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9110 crt=42'9110 lcod 42'9109 mlcod 42'9109 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:02.204 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:02.197+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9113 (0'0,42'9113] local-lis/les=41/42 n=3012 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9112 crt=42'9112 lcod 42'9111 mlcod 42'9111 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:02.717 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:02.713+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9122 (0'0,42'9122] local-lis/les=41/42 n=3003 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9119 crt=42'9119 lcod 42'9118 mlcod 42'9118 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:02.794 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:02.788+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9122 (0'0,42'9122] local-lis/les=41/42 n=3003 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9122 lcod 42'9121 mlcod 42'9121 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:02.920 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:02.916+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9122 (0'0,42'9122] local-lis/les=41/42 n=3003 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9122 lcod 42'9121 mlcod 42'9121 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:03.061 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:03.056+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9124 (0'0,42'9124] local-lis/les=41/42 n=3001 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9123 crt=42'9123 lcod 42'9122 mlcod 42'9122 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:03.204 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:03.193+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9124 (0'0,42'9124] local-lis/les=41/42 n=3001 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9124 lcod 42'9123 mlcod 42'9123 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:03.278 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:03.262+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9124 (0'0,42'9124] local-lis/les=41/42 n=3001 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9124 lcod 42'9123 mlcod 42'9123 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:03.488 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:03.479+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9128 (0'0,42'9128] local-lis/les=41/42 n=2997 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9126 crt=42'9126 lcod 42'9125 mlcod 42'9125 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:03.632 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:03.622+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9129 (0'0,42'9129] local-lis/les=41/42 n=2996 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9128 crt=42'9128 lcod 42'9127 mlcod 42'9127 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:03.819 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:03.815+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9131 (0'0,42'9131] local-lis/les=41/42 n=2994 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9131 lcod 42'9130 mlcod 42'9130 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:03.905 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:03.898+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9131 (0'0,42'9131] local-lis/les=41/42 n=2994 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9131 lcod 42'9130 mlcod 42'9130 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:04.064 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:04.050+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9133 (0'0,42'9133] local-lis/les=41/42 n=2992 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9132 crt=42'9131 lcod 42'9131 mlcod 42'9131 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:04.211 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:04.207+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9135 (0'0,42'9135] local-lis/les=41/42 n=2990 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9135 lcod 42'9134 mlcod 42'9134 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:04.368 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:04.357+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9136 (0'0,42'9136] local-lis/les=41/42 n=2989 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 crt=42'9136 lcod 42'9135 mlcod 42'9135 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:04.433 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:04.427+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9138 (0'0,42'9138] local-lis/les=41/42 n=2987 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9136 crt=42'9136 lcod 42'9135 mlcod 42'9135 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:04.627 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:04.622+0000 7fc58df88640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9139 (0'0,42'9139] local-lis/les=41/42 n=2986 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9138 crt=42'9138 lcod 42'9137 mlcod 42'9137 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:04.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:04.848 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:04.771+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9141 (0'0,42'9141] local-lis/les=41/42 n=2984 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9140 crt=42'9140 lcod 42'9139 mlcod 42'9139 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:05.089 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:05.086+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9142 (0'0,42'9142] local-lis/les=41/42 n=2983 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9141 crt=42'9141 lcod 42'9140 mlcod 42'9140 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:05.284 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-10T20:27:05.281+0000 7fc589f80640 -1 osd.1 pg_epoch: 42 pg[1.bs0( v 42'9148 (0'0,42'9148] local-lis/les=41/42 n=2977 ec=13/13 lis/c=41/41 les/c/f=42/42/0 sis=41) [1,0,2,3]p1(0) r=0 lpr=41 luod=42'9143 crt=42'9143 lcod 42'9142 mlcod 42'9142 active+clean+scrubbing [ 1.bs0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-10T20:27:05.306 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:05.306 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:05.336 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-10T20:27:04.044465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":399307784,"num_objects":48748,"num_object_clones":0,"num_object_copies":194992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125597,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2929,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145723,"ondisk_log_size":145723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725472,"kb_used_data":783304,"kb_used_omap":14,"kb_used_meta":1942001,"kb_avail":374761888,"statfs":{"total":386547056640,"available":383756173312,"internally_reserved":0,"allocated":802103296,"data_stored":801227425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1988609590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,7,8,16,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14712832,"num_objects":-1796,"num_object_clones":0,"num_object_copies":-7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1796,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002572"},"pg_stats":[{"pgid":"1.f","version":"42'9081","reported_seq":9367,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.574288+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:02.574288+0000","last_peered":"2024-09-10T20:27:02.574288+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:02.574288+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9081,"log_dups_size":0,"ondisk_log_size":9081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24911872,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"41'9174","reported_seq":9308,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-10T20:27:00.179142+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:00.179142+0000","last_peered":"2024-09-10T20:27:00.179142+0000","last_clean":"2024-09-10T20:27:00.179142+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:00.179142+0000","last_undegraded":"2024-09-10T20:27:00.179142+0000","last_fullsized":"2024-09-10T20:27:00.179142+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25214976,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'9055","reported_seq":9258,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.345268+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:59.345268+0000","last_peered":"2024-09-10T20:26:59.345268+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:59.345268+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25270888,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9054,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"41'9271","reported_seq":13442,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-10T20:27:00.333650+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:00.333650+0000","last_peered":"2024-09-10T20:27:00.333650+0000","last_clean":"2024-09-10T20:27:00.333650+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:00.333650+0000","last_undegraded":"2024-09-10T20:27:00.333650+0000","last_fullsized":"2024-09-10T20:27:00.333650+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25681920,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9160","reported_seq":9482,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.674501+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:02.674501+0000","last_peered":"2024-09-10T20:27:02.674501+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:02.674501+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24646816,"num_objects":3009,"num_object_clones":0,"num_object_copies":12036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9133","reported_seq":9359,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.554878+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:02.554878+0000","last_peered":"2024-09-10T20:27:02.554878+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:02.554878+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9133,"log_dups_size":0,"ondisk_log_size":9133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24289280,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9133,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9090","reported_seq":9297,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:01.647857+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:01.647857+0000","last_peered":"2024-09-10T20:27:01.647857+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:01.647857+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409248,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9088,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1087}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9221","reported_seq":9393,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:01.690491+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:01.690491+0000","last_peered":"2024-09-10T20:27:01.690491+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:01.690491+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9221,"log_dups_size":0,"ondisk_log_size":9221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25162912,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9009","reported_seq":12985,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:02.454908+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:02.454908+0000","last_peered":"2024-09-10T20:27:02.454908+0000","last_clean":"2024-09-10T20:27:02.454908+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:02.454908+0000","last_undegraded":"2024-09-10T20:27:02.454908+0000","last_fullsized":"2024-09-10T20:27:02.454908+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24509008,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5562,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8991","reported_seq":12961,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:59.390238+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:59.390238+0000","last_peered":"2024-09-10T20:26:59.390238+0000","last_clean":"2024-09-10T20:26:59.390238+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:59.390238+0000","last_undegraded":"2024-09-10T20:26:59.390238+0000","last_fullsized":"2024-09-10T20:26:59.390238+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25812992,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9064","reported_seq":9341,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.575359+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:02.575359+0000","last_peered":"2024-09-10T20:27:02.575359+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:02.575359+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9064,"log_dups_size":0,"ondisk_log_size":9064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24973040,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9166","reported_seq":9468,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.719873+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:02.719873+0000","last_peered":"2024-09-10T20:27:02.719873+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:02.719873+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24566352,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9165,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8839","reported_seq":9022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.286332+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:59.286332+0000","last_peered":"2024-09-10T20:26:59.286332+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:59.286332+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24509008,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9108","reported_seq":9458,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.725105+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:02.725105+0000","last_peered":"2024-09-10T20:27:02.725105+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:02.725105+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24458400,"num_objects":2986,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1148}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9276","reported_seq":9480,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.412775+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:59.412775+0000","last_peered":"2024-09-10T20:26:59.412775+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:59.412775+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990304,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'9085","reported_seq":13135,"reported_epoch":41,"state":"peering","last_fresh":"2024-09-10T20:26:59.492624+0000","last_change":"2024-09-10T20:27:00.423930+0000","last_active":"2024-09-10T20:27:00.423930+0000","last_peered":"2024-09-10T20:26:59.492624+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:59.492624+0000","last_undegraded":"2024-09-10T20:26:59.492624+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24900768,"num_objects":3040,"num_object_clones":0,"num_object_copies":12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9083,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1983,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":399307784,"num_objects":48748,"num_object_clones":0,"num_object_copies":194992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125597,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2929,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":799633408,"data_stored":799633408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145723,"ondisk_log_size":145723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707660,"kb_used_data":194236,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93664180,"statfs":{"total":96636764160,"available":95912120320,"internally_reserved":0,"allocated":198897664,"data_stored":198675148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":525725392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693880,"kb_used_data":197576,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93677960,"statfs":{"total":96636764160,"available":95926231040,"internally_reserved":0,"allocated":202317824,"data_stored":202104763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615472,"kb_used_data":196440,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93756368,"statfs":{"total":96636764160,"available":96006520832,"internally_reserved":0,"allocated":201154560,"data_stored":200936782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708460,"kb_used_data":195052,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93663380,"statfs":{"total":96636764160,"available":95911301120,"internally_reserved":0,"allocated":199733248,"data_stored":199510732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":525659855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199360512,"data_stored":199360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200654848,"data_stored":200654848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201166848,"data_stored":201166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":198451200,"data_stored":198451200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:05.337 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:05.935 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:05.935 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:05.960 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-10T20:27:04.044465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":399307784,"num_objects":48748,"num_object_clones":0,"num_object_copies":194992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125597,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2929,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145723,"ondisk_log_size":145723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725472,"kb_used_data":783304,"kb_used_omap":14,"kb_used_meta":1942001,"kb_avail":374761888,"statfs":{"total":386547056640,"available":383756173312,"internally_reserved":0,"allocated":802103296,"data_stored":801227425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1988609590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,7,8,16,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14712832,"num_objects":-1796,"num_object_clones":0,"num_object_copies":-7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1796,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002572"},"pg_stats":[{"pgid":"1.f","version":"42'9081","reported_seq":9367,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.574288+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:02.574288+0000","last_peered":"2024-09-10T20:27:02.574288+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:02.574288+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9081,"log_dups_size":0,"ondisk_log_size":9081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24911872,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"41'9174","reported_seq":9308,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-10T20:27:00.179142+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:00.179142+0000","last_peered":"2024-09-10T20:27:00.179142+0000","last_clean":"2024-09-10T20:27:00.179142+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:00.179142+0000","last_undegraded":"2024-09-10T20:27:00.179142+0000","last_fullsized":"2024-09-10T20:27:00.179142+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25214976,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'9055","reported_seq":9258,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.345268+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:26:59.345268+0000","last_peered":"2024-09-10T20:26:59.345268+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:26:59.345268+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25270888,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9054,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"41'9271","reported_seq":13442,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-10T20:27:00.333650+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:00.333650+0000","last_peered":"2024-09-10T20:27:00.333650+0000","last_clean":"2024-09-10T20:27:00.333650+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:00.333650+0000","last_undegraded":"2024-09-10T20:27:00.333650+0000","last_fullsized":"2024-09-10T20:27:00.333650+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25681920,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9160","reported_seq":9482,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.674501+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:02.674501+0000","last_peered":"2024-09-10T20:27:02.674501+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:02.674501+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24646816,"num_objects":3009,"num_object_clones":0,"num_object_copies":12036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9133","reported_seq":9359,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.554878+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:02.554878+0000","last_peered":"2024-09-10T20:27:02.554878+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:02.554878+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9133,"log_dups_size":0,"ondisk_log_size":9133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24289280,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9133,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9090","reported_seq":9297,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:01.647857+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:01.647857+0000","last_peered":"2024-09-10T20:27:01.647857+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:01.647857+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409248,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9088,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1087}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9221","reported_seq":9393,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:01.690491+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:01.690491+0000","last_peered":"2024-09-10T20:27:01.690491+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:01.690491+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9221,"log_dups_size":0,"ondisk_log_size":9221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25162912,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9009","reported_seq":12985,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:02.454908+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:02.454908+0000","last_peered":"2024-09-10T20:27:02.454908+0000","last_clean":"2024-09-10T20:27:02.454908+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:02.454908+0000","last_undegraded":"2024-09-10T20:27:02.454908+0000","last_fullsized":"2024-09-10T20:27:02.454908+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24509008,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5562,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'8991","reported_seq":12961,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-10T20:26:59.390238+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:26:59.390238+0000","last_peered":"2024-09-10T20:26:59.390238+0000","last_clean":"2024-09-10T20:26:59.390238+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:26:59.390238+0000","last_undegraded":"2024-09-10T20:26:59.390238+0000","last_fullsized":"2024-09-10T20:26:59.390238+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25812992,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9064","reported_seq":9341,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.575359+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:02.575359+0000","last_peered":"2024-09-10T20:27:02.575359+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:02.575359+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9064,"log_dups_size":0,"ondisk_log_size":9064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24973040,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9166","reported_seq":9468,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.719873+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:02.719873+0000","last_peered":"2024-09-10T20:27:02.719873+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:02.719873+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24566352,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9165,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'8839","reported_seq":9022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.286332+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:26:59.286332+0000","last_peered":"2024-09-10T20:26:59.286332+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:26:59.286332+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24509008,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9108","reported_seq":9458,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.725105+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:02.725105+0000","last_peered":"2024-09-10T20:27:02.725105+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:02.725105+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24458400,"num_objects":2986,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1148}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'9276","reported_seq":9480,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:26:59.412775+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:26:59.412775+0000","last_peered":"2024-09-10T20:26:59.412775+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:26:59.412775+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990304,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"40'9085","reported_seq":13135,"reported_epoch":41,"state":"peering","last_fresh":"2024-09-10T20:26:59.492624+0000","last_change":"2024-09-10T20:27:00.423930+0000","last_active":"2024-09-10T20:27:00.423930+0000","last_peered":"2024-09-10T20:26:59.492624+0000","last_clean":"2024-09-10T20:19:05.527507+0000","last_became_active":"2024-09-10T20:20:16.137754+0000","last_became_peered":"2024-09-10T20:20:16.137754+0000","last_unstale":"2024-09-10T20:26:59.492624+0000","last_undegraded":"2024-09-10T20:26:59.492624+0000","last_fullsized":"2024-09-10T20:20:15.464073+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24900768,"num_objects":3040,"num_object_clones":0,"num_object_copies":12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9083,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1983,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":399307784,"num_objects":48748,"num_object_clones":0,"num_object_copies":194992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125597,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2929,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":799633408,"data_stored":799633408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145723,"ondisk_log_size":145723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707660,"kb_used_data":194236,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93664180,"statfs":{"total":96636764160,"available":95912120320,"internally_reserved":0,"allocated":198897664,"data_stored":198675148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":525725392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693880,"kb_used_data":197576,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93677960,"statfs":{"total":96636764160,"available":95926231040,"internally_reserved":0,"allocated":202317824,"data_stored":202104763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615472,"kb_used_data":196440,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93756368,"statfs":{"total":96636764160,"available":96006520832,"internally_reserved":0,"allocated":201154560,"data_stored":200936782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708460,"kb_used_data":195052,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93663380,"statfs":{"total":96636764160,"available":95911301120,"internally_reserved":0,"allocated":199733248,"data_stored":199510732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":525659855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199360512,"data_stored":199360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200654848,"data_stored":200654848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201166848,"data_stored":201166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":198451200,"data_stored":198451200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:05.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9081", 'reported_seq': 9367, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:02.574288+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:02.574288+0000', 'last_peered': '2024-09-10T20:27:02.574288+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:02.574288+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9081, 'log_dups_size': 0, 'ondisk_log_size': 9081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24911872, 'num_objects': 3041, 'num_object_clones': 0, 'num_object_copies': 12164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9081, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'9055", 'reported_seq': 9258, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:59.345268+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:26:59.345268+0000', 'last_peered': '2024-09-10T20:26:59.345268+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:26:59.345268+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9055, 'log_dups_size': 0, 'ondisk_log_size': 9055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25270888, 'num_objects': 3086, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3086, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9054, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1178}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9160", 'reported_seq': 9482, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:02.674501+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:02.674501+0000', 'last_peered': '2024-09-10T20:27:02.674501+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:02.674501+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9160, 'log_dups_size': 0, 'ondisk_log_size': 9160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24646816, 'num_objects': 3009, 'num_object_clones': 0, 'num_object_copies': 12036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9158, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1154}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9133", 'reported_seq': 9359, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:02.554878+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:02.554878+0000', 'last_peered': '2024-09-10T20:27:02.554878+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:02.554878+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9133, 'log_dups_size': 0, 'ondisk_log_size': 9133, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24289280, 'num_objects': 2965, 'num_object_clones': 0, 'num_object_copies': 11860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9133, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1108}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9090", 'reported_seq': 9297, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:01.647857+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:01.647857+0000', 'last_peered': '2024-09-10T20:27:01.647857+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:01.647857+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9090, 'log_dups_size': 0, 'ondisk_log_size': 9090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24409248, 'num_objects': 2980, 'num_object_clones': 0, 'num_object_copies': 11920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9088, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1087}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:05.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9221", 'reported_seq': 9393, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:01.690491+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:01.690491+0000', 'last_peered': '2024-09-10T20:27:01.690491+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:01.690491+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9221, 'log_dups_size': 0, 'ondisk_log_size': 9221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25162912, 'num_objects': 3072, 'num_object_clones': 0, 'num_object_copies': 12288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9219, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9064", 'reported_seq': 9341, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:02.575359+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:02.575359+0000', 'last_peered': '2024-09-10T20:27:02.575359+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:02.575359+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9064, 'log_dups_size': 0, 'ondisk_log_size': 9064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24973040, 'num_objects': 3049, 'num_object_clones': 0, 'num_object_copies': 12196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9061, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1193}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9166", 'reported_seq': 9468, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:02.719873+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:02.719873+0000', 'last_peered': '2024-09-10T20:27:02.719873+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:02.719873+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9166, 'log_dups_size': 0, 'ondisk_log_size': 9166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24566352, 'num_objects': 2999, 'num_object_clones': 0, 'num_object_copies': 11996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9165, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1139}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'8839", 'reported_seq': 9022, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:59.286332+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:26:59.286332+0000', 'last_peered': '2024-09-10T20:26:59.286332+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:26:59.286332+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8839, 'log_dups_size': 0, 'ondisk_log_size': 8839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24509008, 'num_objects': 2992, 'num_object_clones': 0, 'num_object_copies': 11968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8838, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9108", 'reported_seq': 9458, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:02.725105+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:02.725105+0000', 'last_peered': '2024-09-10T20:27:02.725105+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:02.725105+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9108, 'log_dups_size': 0, 'ondisk_log_size': 9108, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24458400, 'num_objects': 2986, 'num_object_clones': 0, 'num_object_copies': 11944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1838, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9106, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 141, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "40'9276", 'reported_seq': 9480, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:26:59.412775+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:26:59.412775+0000', 'last_peered': '2024-09-10T20:26:59.412775+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:26:59.412775+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9276, 'log_dups_size': 0, 'ondisk_log_size': 9276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25990304, 'num_objects': 3173, 'num_object_clones': 0, 'num_object_copies': 12692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9274, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:27:05.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'9085", 'reported_seq': 13135, 'reported_epoch': 41, 'state': 'peering', 'last_fresh': '2024-09-10T20:26:59.492624+0000', 'last_change': '2024-09-10T20:27:00.423930+0000', 'last_active': '2024-09-10T20:27:00.423930+0000', 'last_peered': '2024-09-10T20:26:59.492624+0000', 'last_clean': '2024-09-10T20:19:05.527507+0000', 'last_became_active': '2024-09-10T20:20:16.137754+0000', 'last_became_peered': '2024-09-10T20:20:16.137754+0000', 'last_unstale': '2024-09-10T20:26:59.492624+0000', 'last_undegraded': '2024-09-10T20:26:59.492624+0000', 'last_fullsized': '2024-09-10T20:20:15.464073+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9085, 'log_dups_size': 0, 'ondisk_log_size': 9085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24900768, 'num_objects': 3040, 'num_object_clones': 0, 'num_object_copies': 12160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9083, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1983, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-10T20:27:05.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:06.695 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:06.696 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20747/194992 objects degraded (10.640%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":515,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":4},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":48748,"data_bytes":399307784,"bytes_used":2790883328,"bytes_avail":383756173312,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":20747,"degraded_total":194992,"degraded_ratio":0.10639923689176993,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":224},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:27:06.727 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:06.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:07.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:07.333 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:07.355 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-10T20:27:06.045028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":395195400,"num_objects":48246,"num_object_clones":0,"num_object_copies":192984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126099,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2929,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146225,"ondisk_log_size":146225,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743068,"kb_used_data":774572,"kb_used_omap":14,"kb_used_meta":1968369,"kb_avail":374744292,"statfs":{"total":386547056640,"available":383738155008,"internally_reserved":0,"allocated":793161728,"data_stored":792271664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2015610422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,5,6,13,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12845056,"num_objects":-1568,"num_object_clones":0,"num_object_copies":-6272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1568,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002643"},"pg_stats":[{"pgid":"1.f","version":"42'9081","reported_seq":9367,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.574288+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:02.574288+0000","last_peered":"2024-09-10T20:27:02.574288+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:02.574288+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9081,"log_dups_size":0,"ondisk_log_size":9081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24911872,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9254","reported_seq":9390,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:05.502911+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:05.502911+0000","last_peered":"2024-09-10T20:27:05.502911+0000","last_clean":"2024-09-10T20:27:05.502911+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:05.502911+0000","last_undegraded":"2024-09-10T20:27:05.502911+0000","last_fullsized":"2024-09-10T20:27:05.502911+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9254,"log_dups_size":0,"ondisk_log_size":9254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24559616,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9126","reported_seq":9332,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:04.455602+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:04.455602+0000","last_peered":"2024-09-10T20:27:04.455602+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:04.455602+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24689256,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9125,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9357","reported_seq":13530,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:05.519452+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:05.519452+0000","last_peered":"2024-09-10T20:27:05.519452+0000","last_clean":"2024-09-10T20:27:05.519452+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:05.519452+0000","last_undegraded":"2024-09-10T20:27:05.519452+0000","last_fullsized":"2024-09-10T20:27:05.519452+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9160","reported_seq":9482,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.674501+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:02.674501+0000","last_peered":"2024-09-10T20:27:02.674501+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:02.674501+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24646816,"num_objects":3009,"num_object_clones":0,"num_object_copies":12036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9133","reported_seq":9359,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.554878+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:02.554878+0000","last_peered":"2024-09-10T20:27:02.554878+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:02.554878+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9133,"log_dups_size":0,"ondisk_log_size":9133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24289280,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9133,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9090","reported_seq":9297,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:01.647857+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:01.647857+0000","last_peered":"2024-09-10T20:27:01.647857+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:01.647857+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409248,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9088,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1087}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9221","reported_seq":9393,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:01.690491+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:01.690491+0000","last_peered":"2024-09-10T20:27:01.690491+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:01.690491+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9221,"log_dups_size":0,"ondisk_log_size":9221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25162912,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9009","reported_seq":12985,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:02.454908+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:02.454908+0000","last_peered":"2024-09-10T20:27:02.454908+0000","last_clean":"2024-09-10T20:27:02.454908+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:02.454908+0000","last_undegraded":"2024-09-10T20:27:02.454908+0000","last_fullsized":"2024-09-10T20:27:02.454908+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24509008,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5562,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9054","reported_seq":13028,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:04.433556+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:04.433556+0000","last_peered":"2024-09-10T20:27:04.433556+0000","last_clean":"2024-09-10T20:27:04.433556+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:04.433556+0000","last_undegraded":"2024-09-10T20:27:04.433556+0000","last_fullsized":"2024-09-10T20:27:04.433556+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9054,"log_dups_size":0,"ondisk_log_size":9054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1692,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9064","reported_seq":9341,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.575359+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:02.575359+0000","last_peered":"2024-09-10T20:27:02.575359+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:02.575359+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9064,"log_dups_size":0,"ondisk_log_size":9064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24973040,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9166","reported_seq":9468,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.719873+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:02.719873+0000","last_peered":"2024-09-10T20:27:02.719873+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:02.719873+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24566352,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9165,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'8909","reported_seq":9096,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:04.437708+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:04.437708+0000","last_peered":"2024-09-10T20:27:04.437708+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:04.437708+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23935568,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9108","reported_seq":9458,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:02.725105+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:02.725105+0000","last_peered":"2024-09-10T20:27:02.725105+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:02.725105+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24458400,"num_objects":2986,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1148}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9342","reported_seq":9551,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:04.419870+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:04.419870+0000","last_peered":"2024-09-10T20:27:04.419870+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:04.419870+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9342,"log_dups_size":0,"ondisk_log_size":9342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25449632,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9340,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1199}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9151","reported_seq":13245,"reported_epoch":42,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:27:05.428217+0000","last_change":"2024-09-10T20:27:01.198931+0000","last_active":"2024-09-10T20:27:05.428217+0000","last_peered":"2024-09-10T20:27:05.428217+0000","last_clean":"2024-09-10T20:27:05.428217+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:05.428217+0000","last_undegraded":"2024-09-10T20:27:05.428217+0000","last_fullsized":"2024-09-10T20:27:05.428217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":2901,"log_size":9151,"log_dups_size":0,"ondisk_log_size":9151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360096,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9149,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1983,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":395195400,"num_objects":48246,"num_object_clones":0,"num_object_copies":192984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126099,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2929,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":790495232,"data_stored":790495232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146225,"ondisk_log_size":146225,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707660,"kb_used_data":194236,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93664180,"statfs":{"total":96636764160,"available":95912120320,"internally_reserved":0,"allocated":198897664,"data_stored":198675148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":525725392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693628,"kb_used_data":193196,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93678212,"statfs":{"total":96636764160,"available":95926489088,"internally_reserved":0,"allocated":197832704,"data_stored":197610188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633320,"kb_used_data":192088,"kb_used_omap":4,"kb_used_meta":441211,"kb_avail":93738520,"statfs":{"total":96636764160,"available":95988244480,"internally_reserved":0,"allocated":196698112,"data_stored":196475596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451800786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708460,"kb_used_data":195052,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93663380,"statfs":{"total":96636764160,"available":95911301120,"internally_reserved":0,"allocated":199733248,"data_stored":199510732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":525659855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199360512,"data_stored":199360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195923968,"data_stored":195923968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196759552,"data_stored":196759552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":198451200,"data_stored":198451200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:10.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:10.920 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:10.920 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:10.940 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-10T20:27:10.046077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":387773448,"num_objects":47340,"num_object_clones":0,"num_object_copies":189360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47340,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127005,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2970,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147131,"ondisk_log_size":147131,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709992,"kb_used_data":762320,"kb_used_omap":14,"kb_used_meta":1947569,"kb_avail":374777368,"statfs":{"total":386547056640,"available":383772024832,"internally_reserved":0,"allocated":780615680,"data_stored":779725616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1994311222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,4,2,10,7,9,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18087936,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003042"},"pg_stats":[{"pgid":"1.f","version":"42'9154","reported_seq":9439,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.773245+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:07.773245+0000","last_peered":"2024-09-10T20:27:07.773245+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:07.773245+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313856,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9254","reported_seq":9390,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:05.502911+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:05.502911+0000","last_peered":"2024-09-10T20:27:05.502911+0000","last_clean":"2024-09-10T20:27:05.502911+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:05.502911+0000","last_undegraded":"2024-09-10T20:27:05.502911+0000","last_fullsized":"2024-09-10T20:27:05.502911+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9254,"log_dups_size":0,"ondisk_log_size":9254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24559616,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9194","reported_seq":9401,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.413053+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:09.413053+0000","last_peered":"2024-09-10T20:27:09.413053+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:09.413053+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24132200,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9193,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9357","reported_seq":13530,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:05.519452+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:05.519452+0000","last_peered":"2024-09-10T20:27:05.519452+0000","last_clean":"2024-09-10T20:27:05.519452+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:05.519452+0000","last_undegraded":"2024-09-10T20:27:05.519452+0000","last_fullsized":"2024-09-10T20:27:05.519452+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9232","reported_seq":9554,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.711246+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:07.711246+0000","last_peered":"2024-09-10T20:27:07.711246+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:07.711246+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24056992,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9230,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9213","reported_seq":9437,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.853669+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:07.853669+0000","last_peered":"2024-09-10T20:27:07.853669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:07.853669+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23633920,"num_objects":2885,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9162","reported_seq":9369,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:06.674836+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:06.674836+0000","last_peered":"2024-09-10T20:27:06.674836+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:06.674836+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23819424,"num_objects":2908,"num_object_clones":0,"num_object_copies":11632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9160,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1015}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9290","reported_seq":9462,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:06.763630+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:06.763630+0000","last_peered":"2024-09-10T20:27:06.763630+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:06.763630+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24597664,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9288,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9079","reported_seq":13055,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:07.687615+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:07.687615+0000","last_peered":"2024-09-10T20:27:07.687615+0000","last_clean":"2024-09-10T20:27:07.687615+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:07.687615+0000","last_undegraded":"2024-09-10T20:27:07.687615+0000","last_fullsized":"2024-09-10T20:27:07.687615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23935568,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5632,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9119","reported_seq":13094,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:09.619656+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:09.619656+0000","last_peered":"2024-09-10T20:27:09.619656+0000","last_clean":"2024-09-10T20:27:09.619656+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:09.619656+0000","last_undegraded":"2024-09-10T20:27:09.619656+0000","last_fullsized":"2024-09-10T20:27:09.619656+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9119,"log_dups_size":0,"ondisk_log_size":9119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24764416,"num_objects":3023,"num_object_clones":0,"num_object_copies":12092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9128","reported_seq":9404,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.793338+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:07.793338+0000","last_peered":"2024-09-10T20:27:07.793338+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:07.793338+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24448752,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9226","reported_seq":9528,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.701652+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:07.701652+0000","last_peered":"2024-09-10T20:27:07.701652+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:07.701652+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24074832,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'8976","reported_seq":9163,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.551641+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:09.551641+0000","last_peered":"2024-09-10T20:27:09.551641+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:09.551641+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386704,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9179","reported_seq":9613,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.714813+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:07.714813+0000","last_peered":"2024-09-10T20:27:07.714813+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:07.714813+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9179,"log_dups_size":0,"ondisk_log_size":9179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23876768,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9177,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9417","reported_seq":9626,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.559504+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:09.559504+0000","last_peered":"2024-09-10T20:27:09.559504+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:09.559504+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9417,"log_dups_size":0,"ondisk_log_size":9417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9415,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9151","reported_seq":13245,"reported_epoch":42,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:27:05.428217+0000","last_change":"2024-09-10T20:27:01.198931+0000","last_active":"2024-09-10T20:27:05.428217+0000","last_peered":"2024-09-10T20:27:05.428217+0000","last_clean":"2024-09-10T20:27:05.428217+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:05.428217+0000","last_undegraded":"2024-09-10T20:27:05.428217+0000","last_fullsized":"2024-09-10T20:27:05.428217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":2901,"log_size":9151,"log_dups_size":0,"ondisk_log_size":9151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360096,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9149,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1983,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":387773448,"num_objects":47340,"num_object_clones":0,"num_object_copies":189360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47340,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127005,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2970,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":776830976,"data_stored":776830976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147131,"ondisk_log_size":147131,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117089,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690920,"kb_used_data":189976,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93680920,"statfs":{"total":96636764160,"available":95929262080,"internally_reserved":0,"allocated":194535424,"data_stored":194312908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,4,1,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693436,"kb_used_data":188832,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93678404,"statfs":{"total":96636764160,"available":95926685696,"internally_reserved":0,"allocated":193363968,"data_stored":193141452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633320,"kb_used_data":192088,"kb_used_omap":4,"kb_used_meta":441211,"kb_avail":93738520,"statfs":{"total":96636764160,"available":95988244480,"internally_reserved":0,"allocated":196698112,"data_stored":196475596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451800786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215202,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692316,"kb_used_data":191424,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93679524,"statfs":{"total":96636764160,"available":95927832576,"internally_reserved":0,"allocated":196018176,"data_stored":195795660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,0,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194830336,"data_stored":194830336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195923968,"data_stored":195923968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192229376,"data_stored":192229376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193847296,"data_stored":193847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:10.942 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:11.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:11.577 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:11.601 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-10T20:27:10.046077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":387773448,"num_objects":47340,"num_object_clones":0,"num_object_copies":189360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47340,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127005,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2970,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147131,"ondisk_log_size":147131,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709992,"kb_used_data":762320,"kb_used_omap":14,"kb_used_meta":1947569,"kb_avail":374777368,"statfs":{"total":386547056640,"available":383772024832,"internally_reserved":0,"allocated":780615680,"data_stored":779725616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1994311222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,4,2,10,7,9,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18087936,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003042"},"pg_stats":[{"pgid":"1.f","version":"42'9154","reported_seq":9439,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.773245+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:07.773245+0000","last_peered":"2024-09-10T20:27:07.773245+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:07.773245+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313856,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9254","reported_seq":9390,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:05.502911+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:05.502911+0000","last_peered":"2024-09-10T20:27:05.502911+0000","last_clean":"2024-09-10T20:27:05.502911+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:05.502911+0000","last_undegraded":"2024-09-10T20:27:05.502911+0000","last_fullsized":"2024-09-10T20:27:05.502911+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9254,"log_dups_size":0,"ondisk_log_size":9254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24559616,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9194","reported_seq":9401,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.413053+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:09.413053+0000","last_peered":"2024-09-10T20:27:09.413053+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:09.413053+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24132200,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9193,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9357","reported_seq":13530,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:05.519452+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:05.519452+0000","last_peered":"2024-09-10T20:27:05.519452+0000","last_clean":"2024-09-10T20:27:05.519452+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:05.519452+0000","last_undegraded":"2024-09-10T20:27:05.519452+0000","last_fullsized":"2024-09-10T20:27:05.519452+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9232","reported_seq":9554,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.711246+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:07.711246+0000","last_peered":"2024-09-10T20:27:07.711246+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:07.711246+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24056992,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9230,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9213","reported_seq":9437,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.853669+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:07.853669+0000","last_peered":"2024-09-10T20:27:07.853669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:07.853669+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23633920,"num_objects":2885,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9162","reported_seq":9369,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:06.674836+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:06.674836+0000","last_peered":"2024-09-10T20:27:06.674836+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:06.674836+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23819424,"num_objects":2908,"num_object_clones":0,"num_object_copies":11632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9160,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1015}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9290","reported_seq":9462,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:06.763630+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:06.763630+0000","last_peered":"2024-09-10T20:27:06.763630+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:06.763630+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24597664,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9288,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9079","reported_seq":13055,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:07.687615+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:07.687615+0000","last_peered":"2024-09-10T20:27:07.687615+0000","last_clean":"2024-09-10T20:27:07.687615+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:07.687615+0000","last_undegraded":"2024-09-10T20:27:07.687615+0000","last_fullsized":"2024-09-10T20:27:07.687615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23935568,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5632,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9119","reported_seq":13094,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:09.619656+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:09.619656+0000","last_peered":"2024-09-10T20:27:09.619656+0000","last_clean":"2024-09-10T20:27:09.619656+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:09.619656+0000","last_undegraded":"2024-09-10T20:27:09.619656+0000","last_fullsized":"2024-09-10T20:27:09.619656+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9119,"log_dups_size":0,"ondisk_log_size":9119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24764416,"num_objects":3023,"num_object_clones":0,"num_object_copies":12092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9128","reported_seq":9404,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.793338+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:07.793338+0000","last_peered":"2024-09-10T20:27:07.793338+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:07.793338+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24448752,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9226","reported_seq":9528,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.701652+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:07.701652+0000","last_peered":"2024-09-10T20:27:07.701652+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:07.701652+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24074832,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'8976","reported_seq":9163,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.551641+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:09.551641+0000","last_peered":"2024-09-10T20:27:09.551641+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:09.551641+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386704,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9179","reported_seq":9613,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.714813+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:07.714813+0000","last_peered":"2024-09-10T20:27:07.714813+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:07.714813+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9179,"log_dups_size":0,"ondisk_log_size":9179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23876768,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9177,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9417","reported_seq":9626,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.559504+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:09.559504+0000","last_peered":"2024-09-10T20:27:09.559504+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:09.559504+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9417,"log_dups_size":0,"ondisk_log_size":9417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9415,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9151","reported_seq":13245,"reported_epoch":42,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:27:05.428217+0000","last_change":"2024-09-10T20:27:01.198931+0000","last_active":"2024-09-10T20:27:05.428217+0000","last_peered":"2024-09-10T20:27:05.428217+0000","last_clean":"2024-09-10T20:27:05.428217+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:05.428217+0000","last_undegraded":"2024-09-10T20:27:05.428217+0000","last_fullsized":"2024-09-10T20:27:05.428217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":2901,"log_size":9151,"log_dups_size":0,"ondisk_log_size":9151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360096,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9149,"num_write_kb":48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1983,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":387773448,"num_objects":47340,"num_object_clones":0,"num_object_copies":189360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47340,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127005,"num_write_kb":627355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2970,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":776830976,"data_stored":776830976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147131,"ondisk_log_size":147131,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117089,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690920,"kb_used_data":189976,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93680920,"statfs":{"total":96636764160,"available":95929262080,"internally_reserved":0,"allocated":194535424,"data_stored":194312908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,4,1,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693436,"kb_used_data":188832,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93678404,"statfs":{"total":96636764160,"available":95926685696,"internally_reserved":0,"allocated":193363968,"data_stored":193141452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633320,"kb_used_data":192088,"kb_used_omap":4,"kb_used_meta":441211,"kb_avail":93738520,"statfs":{"total":96636764160,"available":95988244480,"internally_reserved":0,"allocated":196698112,"data_stored":196475596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451800786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215202,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692316,"kb_used_data":191424,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93679524,"statfs":{"total":96636764160,"available":95927832576,"internally_reserved":0,"allocated":196018176,"data_stored":195795660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,0,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194830336,"data_stored":194830336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195923968,"data_stored":195923968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192229376,"data_stored":192229376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193847296,"data_stored":193847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:11.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:11.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9154", 'reported_seq': 9439, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:07.773245+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:07.773245+0000', 'last_peered': '2024-09-10T20:27:07.773245+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:07.773245+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9154, 'log_dups_size': 0, 'ondisk_log_size': 9154, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24313856, 'num_objects': 2968, 'num_object_clones': 0, 'num_object_copies': 11872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9154, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1113}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9194", 'reported_seq': 9401, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:09.413053+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:09.413053+0000', 'last_peered': '2024-09-10T20:27:09.413053+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:09.413053+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9194, 'log_dups_size': 0, 'ondisk_log_size': 9194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24132200, 'num_objects': 2947, 'num_object_clones': 0, 'num_object_copies': 11788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2947, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9193, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1039}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9232", 'reported_seq': 9554, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:07.711246+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:07.711246+0000', 'last_peered': '2024-09-10T20:27:07.711246+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:07.711246+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9232, 'log_dups_size': 0, 'ondisk_log_size': 9232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24056992, 'num_objects': 2937, 'num_object_clones': 0, 'num_object_copies': 11748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9230, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9213", 'reported_seq': 9437, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:07.853669+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:07.853669+0000', 'last_peered': '2024-09-10T20:27:07.853669+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:07.853669+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9213, 'log_dups_size': 0, 'ondisk_log_size': 9213, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23633920, 'num_objects': 2885, 'num_object_clones': 0, 'num_object_copies': 11540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9213, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1028}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9162", 'reported_seq': 9369, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:06.674836+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:06.674836+0000', 'last_peered': '2024-09-10T20:27:06.674836+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:06.674836+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9162, 'log_dups_size': 0, 'ondisk_log_size': 9162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23819424, 'num_objects': 2908, 'num_object_clones': 0, 'num_object_copies': 11632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9160, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1015}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:11.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9290", 'reported_seq': 9462, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:06.763630+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:06.763630+0000', 'last_peered': '2024-09-10T20:27:06.763630+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:06.763630+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9290, 'log_dups_size': 0, 'ondisk_log_size': 9290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24597664, 'num_objects': 3003, 'num_object_clones': 0, 'num_object_copies': 12012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9288, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9128", 'reported_seq': 9404, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:07.793338+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:07.793338+0000', 'last_peered': '2024-09-10T20:27:07.793338+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:07.793338+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9128, 'log_dups_size': 0, 'ondisk_log_size': 9128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24448752, 'num_objects': 2985, 'num_object_clones': 0, 'num_object_copies': 11940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9125, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1129}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9226", 'reported_seq': 9528, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:07.701652+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:07.701652+0000', 'last_peered': '2024-09-10T20:27:07.701652+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:07.701652+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9226, 'log_dups_size': 0, 'ondisk_log_size': 9226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24074832, 'num_objects': 2939, 'num_object_clones': 0, 'num_object_copies': 11756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9225, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'8976", 'reported_seq': 9163, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:09.551641+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:09.551641+0000', 'last_peered': '2024-09-10T20:27:09.551641+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:09.551641+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 8976, 'log_dups_size': 0, 'ondisk_log_size': 8976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23386704, 'num_objects': 2855, 'num_object_clones': 0, 'num_object_copies': 11420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8975, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9179", 'reported_seq': 9613, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:07.714813+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:07.714813+0000', 'last_peered': '2024-09-10T20:27:07.714813+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:07.714813+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9179, 'log_dups_size': 0, 'ondisk_log_size': 9179, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23876768, 'num_objects': 2915, 'num_object_clones': 0, 'num_object_copies': 11660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1797, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9177, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 182, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:11.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9417", 'reported_seq': 9626, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:09.559504+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:09.559504+0000', 'last_peered': '2024-09-10T20:27:09.559504+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:09.559504+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9417, 'log_dups_size': 0, 'ondisk_log_size': 9417, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24835232, 'num_objects': 3032, 'num_object_clones': 0, 'num_object_copies': 12128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9415, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:11.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-10T20:27:11.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "42'9151", 'reported_seq': 13245, 'reported_epoch': 42, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-10T20:27:05.428217+0000', 'last_change': '2024-09-10T20:27:01.198931+0000', 'last_active': '2024-09-10T20:27:05.428217+0000', 'last_peered': '2024-09-10T20:27:05.428217+0000', 'last_clean': '2024-09-10T20:27:05.428217+0000', 'last_became_active': '2024-09-10T20:27:00.553749+0000', 'last_became_peered': '2024-09-10T20:27:00.553749+0000', 'last_unstale': '2024-09-10T20:27:05.428217+0000', 'last_undegraded': '2024-09-10T20:27:05.428217+0000', 'last_fullsized': '2024-09-10T20:27:05.428217+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 42, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 2901, 'log_size': 9151, 'log_dups_size': 0, 'ondisk_log_size': 9151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 4s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24360096, 'num_objects': 2974, 'num_object_clones': 0, 'num_object_copies': 11896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9149, 'num_write_kb': 48492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1983, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-10T20:27:11.605 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:12.318 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:12.318 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20706/189360 objects degraded (10.935%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":521,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":4},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47340,"data_bytes":387773448,"bytes_used":2775031808,"bytes_avail":383772024832,"bytes_total":386547056640,"degraded_objects":20706,"degraded_total":189360,"degraded_ratio":0.10934727503168568,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:27:12.352 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:12.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:12.987 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:13.014 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-10T20:27:12.046694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":385130344,"num_objects":47017,"num_object_clones":0,"num_object_copies":188068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47017,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118201,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147452,"ondisk_log_size":147452,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691996,"kb_used_data":752636,"kb_used_omap":14,"kb_used_meta":1939249,"kb_avail":374795364,"statfs":{"total":386547056640,"available":383790452736,"internally_reserved":0,"allocated":770699264,"data_stored":769809200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1985791542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,23,12,13,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19231904,"num_objects":-2348,"num_object_clones":0,"num_object_copies":-9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6779,"num_write_kb":-48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1903,"num_bytes_recovered":-4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.522321"},"pg_stats":[{"pgid":"1.f","version":"42'9154","reported_seq":9439,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.773245+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:07.773245+0000","last_peered":"2024-09-10T20:27:07.773245+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:07.773245+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313856,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9308","reported_seq":9444,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:10.402498+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:10.402498+0000","last_peered":"2024-09-10T20:27:10.402498+0000","last_clean":"2024-09-10T20:27:10.402498+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:10.402498+0000","last_undegraded":"2024-09-10T20:27:10.402498+0000","last_fullsized":"2024-09-10T20:27:10.402498+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9308,"log_dups_size":0,"ondisk_log_size":9308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5749,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9194","reported_seq":9401,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.413053+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:09.413053+0000","last_peered":"2024-09-10T20:27:09.413053+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:09.413053+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24132200,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9193,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9439","reported_seq":13612,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:10.579812+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:10.579812+0000","last_peered":"2024-09-10T20:27:10.579812+0000","last_clean":"2024-09-10T20:27:10.579812+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:10.579812+0000","last_undegraded":"2024-09-10T20:27:10.579812+0000","last_fullsized":"2024-09-10T20:27:10.579812+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9439,"log_dups_size":0,"ondisk_log_size":9439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24305664,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9232","reported_seq":9554,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.711246+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:07.711246+0000","last_peered":"2024-09-10T20:27:07.711246+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:07.711246+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24056992,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9230,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9213","reported_seq":9437,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.853669+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:07.853669+0000","last_peered":"2024-09-10T20:27:07.853669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:07.853669+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23633920,"num_objects":2885,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9223","reported_seq":9430,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:11.815125+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:11.815125+0000","last_peered":"2024-09-10T20:27:11.815125+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:11.815125+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9223,"log_dups_size":0,"ondisk_log_size":9223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23319712,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9221,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9357","reported_seq":9529,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:11.966995+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:11.966995+0000","last_peered":"2024-09-10T20:27:11.966995+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:11.966995+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24048800,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9079","reported_seq":13055,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:07.687615+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:07.687615+0000","last_peered":"2024-09-10T20:27:07.687615+0000","last_clean":"2024-09-10T20:27:07.687615+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:07.687615+0000","last_undegraded":"2024-09-10T20:27:07.687615+0000","last_fullsized":"2024-09-10T20:27:07.687615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23935568,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5632,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9119","reported_seq":13094,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:09.619656+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:09.619656+0000","last_peered":"2024-09-10T20:27:09.619656+0000","last_clean":"2024-09-10T20:27:09.619656+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:09.619656+0000","last_undegraded":"2024-09-10T20:27:09.619656+0000","last_fullsized":"2024-09-10T20:27:09.619656+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9119,"log_dups_size":0,"ondisk_log_size":9119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24764416,"num_objects":3023,"num_object_clones":0,"num_object_copies":12092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9128","reported_seq":9404,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.793338+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:07.793338+0000","last_peered":"2024-09-10T20:27:07.793338+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:07.793338+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24448752,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9226","reported_seq":9528,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.701652+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:07.701652+0000","last_peered":"2024-09-10T20:27:07.701652+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:07.701652+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24074832,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'8976","reported_seq":9163,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.551641+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:09.551641+0000","last_peered":"2024-09-10T20:27:09.551641+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:09.551641+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386704,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9179","reported_seq":9613,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:07.714813+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:07.714813+0000","last_peered":"2024-09-10T20:27:07.714813+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:07.714813+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9179,"log_dups_size":0,"ondisk_log_size":9179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23876768,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9177,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9417","reported_seq":9626,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:09.559504+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:09.559504+0000","last_peered":"2024-09-10T20:27:09.559504+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:09.559504+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9417,"log_dups_size":0,"ondisk_log_size":9417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9415,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9208","reported_seq":13308,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:10.420951+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:10.420951+0000","last_peered":"2024-09-10T20:27:10.420951+0000","last_clean":"2024-09-10T20:27:10.420951+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:10.420951+0000","last_undegraded":"2024-09-10T20:27:10.420951+0000","last_fullsized":"2024-09-10T20:27:10.420951+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9208,"log_dups_size":0,"ondisk_log_size":9208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23879680,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":385130344,"num_objects":47017,"num_object_clones":0,"num_object_copies":188068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47017,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118201,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":767533056,"data_stored":767533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147452,"ondisk_log_size":147452,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117089,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690920,"kb_used_data":189976,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93680920,"statfs":{"total":96636764160,"available":95929262080,"internally_reserved":0,"allocated":194535424,"data_stored":194312908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,4,1,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693436,"kb_used_data":188832,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93678404,"statfs":{"total":96636764160,"available":95926685696,"internally_reserved":0,"allocated":193363968,"data_stored":193141452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616448,"kb_used_data":187684,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93755392,"statfs":{"total":96636764160,"available":96005521408,"internally_reserved":0,"allocated":192188416,"data_stored":191965900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":439021266},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691192,"kb_used_data":186144,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93680648,"statfs":{"total":96636764160,"available":95928983552,"internally_reserved":0,"allocated":190611456,"data_stored":190388940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190124032,"data_stored":190124032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191332352,"data_stored":191332352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192229376,"data_stored":192229376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193847296,"data_stored":193847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:16.016 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:16.621 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:16.621 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:16.643 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-10T20:27:16.048347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376897384,"num_objects":46012,"num_object_clones":0,"num_object_copies":184048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46012,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119206,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148457,"ondisk_log_size":148457,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691304,"kb_used_data":739464,"kb_used_omap":14,"kb_used_meta":1951729,"kb_avail":374796056,"statfs":{"total":386547056640,"available":383791161344,"internally_reserved":0,"allocated":757211136,"data_stored":756321072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1998571062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,5,23,11,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22410400,"num_objects":-2736,"num_object_clones":0,"num_object_copies":-10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6391,"num_write_kb":-48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1884,"num_bytes_recovered":-4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003881"},"pg_stats":[{"pgid":"1.f","version":"42'9234","reported_seq":9519,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.979599+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:12.979599+0000","last_peered":"2024-09-10T20:27:12.979599+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:12.979599+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":2888,"num_object_clones":0,"num_object_copies":11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9386","reported_seq":9522,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.576438+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:15.576438+0000","last_peered":"2024-09-10T20:27:15.576438+0000","last_clean":"2024-09-10T20:27:15.576438+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:15.576438+0000","last_undegraded":"2024-09-10T20:27:15.576438+0000","last_fullsized":"2024-09-10T20:27:15.576438+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23478272,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5827,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9264","reported_seq":9471,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.575640+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:14.575640+0000","last_peered":"2024-09-10T20:27:14.575640+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:14.575640+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9264,"log_dups_size":0,"ondisk_log_size":9264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23558760,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9263,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9504","reported_seq":13676,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.760269+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:15.760269+0000","last_peered":"2024-09-10T20:27:15.760269+0000","last_clean":"2024-09-10T20:27:15.760269+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:15.760269+0000","last_undegraded":"2024-09-10T20:27:15.760269+0000","last_fullsized":"2024-09-10T20:27:15.760269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23773184,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9313","reported_seq":9636,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.829807+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:12.829807+0000","last_peered":"2024-09-10T20:27:12.829807+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:12.829807+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9313,"log_dups_size":0,"ondisk_log_size":9313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23393440,"num_objects":2856,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9311,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9273","reported_seq":9499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.790204+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:12.790204+0000","last_peered":"2024-09-10T20:27:12.790204+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:12.790204+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9273,"log_dups_size":0,"ondisk_log_size":9273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23142400,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9273,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9223","reported_seq":9430,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:11.815125+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:11.815125+0000","last_peered":"2024-09-10T20:27:11.815125+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:11.815125+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9223,"log_dups_size":0,"ondisk_log_size":9223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23319712,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9221,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9357","reported_seq":9529,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:11.966995+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:11.966995+0000","last_peered":"2024-09-10T20:27:11.966995+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:11.966995+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24048800,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9150","reported_seq":13126,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:12.957329+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:12.957329+0000","last_peered":"2024-09-10T20:27:12.957329+0000","last_clean":"2024-09-10T20:27:12.957329+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:12.957329+0000","last_undegraded":"2024-09-10T20:27:12.957329+0000","last_fullsized":"2024-09-10T20:27:12.957329+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9150,"log_dups_size":0,"ondisk_log_size":9150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23353936,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9190","reported_seq":13165,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:14.569606+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:14.569606+0000","last_peered":"2024-09-10T20:27:14.569606+0000","last_clean":"2024-09-10T20:27:14.569606+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:14.569606+0000","last_undegraded":"2024-09-10T20:27:14.569606+0000","last_fullsized":"2024-09-10T20:27:14.569606+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":2952,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1828,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9206","reported_seq":9483,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.789029+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:12.789029+0000","last_peered":"2024-09-10T20:27:12.789029+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:12.789029+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9206,"log_dups_size":0,"ondisk_log_size":9206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23809776,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9304","reported_seq":9605,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.825322+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:12.825322+0000","last_peered":"2024-09-10T20:27:12.825322+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:12.825322+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23435856,"num_objects":2861,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9037","reported_seq":9223,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.513358+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:14.513358+0000","last_peered":"2024-09-10T20:27:14.513358+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:14.513358+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9037,"log_dups_size":0,"ondisk_log_size":9037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22886992,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":891}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9249","reported_seq":9798,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.954060+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:12.954060+0000","last_peered":"2024-09-10T20:27:12.954060+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:12.954060+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303328,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9483","reported_seq":9692,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.570791+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:14.570791+0000","last_peered":"2024-09-10T20:27:14.570791+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:14.570791+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24294560,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9481,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9284","reported_seq":13383,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.714168+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:15.714168+0000","last_peered":"2024-09-10T20:27:15.714168+0000","last_clean":"2024-09-10T20:27:15.714168+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:15.714168+0000","last_undegraded":"2024-09-10T20:27:15.714168+0000","last_fullsized":"2024-09-10T20:27:15.714168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23257088,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376897384,"num_objects":46012,"num_object_clones":0,"num_object_copies":184048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46012,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119206,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":753758208,"data_stored":753758208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148457,"ondisk_log_size":148457,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690680,"kb_used_data":185572,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93681160,"statfs":{"total":96636764160,"available":95929507840,"internally_reserved":0,"allocated":190025728,"data_stored":189803212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,6,5,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693284,"kb_used_data":184520,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93678556,"statfs":{"total":96636764160,"available":95926841344,"internally_reserved":0,"allocated":188948480,"data_stored":188725964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616148,"kb_used_data":183228,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93755692,"statfs":{"total":96636764160,"available":96005828608,"internally_reserved":0,"allocated":187625472,"data_stored":187402956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691192,"kb_used_data":186144,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93680648,"statfs":{"total":96636764160,"available":95928983552,"internally_reserved":0,"allocated":190611456,"data_stored":190388940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190124032,"data_stored":190124032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186662912,"data_stored":186662912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187682816,"data_stored":187682816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":189288448,"data_stored":189288448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:16.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:17.222 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:17.223 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:17.243 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-10T20:27:16.048347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376897384,"num_objects":46012,"num_object_clones":0,"num_object_copies":184048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46012,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119206,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148457,"ondisk_log_size":148457,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691304,"kb_used_data":739464,"kb_used_omap":14,"kb_used_meta":1951729,"kb_avail":374796056,"statfs":{"total":386547056640,"available":383791161344,"internally_reserved":0,"allocated":757211136,"data_stored":756321072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1998571062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,5,23,11,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22410400,"num_objects":-2736,"num_object_clones":0,"num_object_copies":-10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6391,"num_write_kb":-48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1884,"num_bytes_recovered":-4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003881"},"pg_stats":[{"pgid":"1.f","version":"42'9234","reported_seq":9519,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.979599+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:12.979599+0000","last_peered":"2024-09-10T20:27:12.979599+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:12.979599+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":2888,"num_object_clones":0,"num_object_copies":11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9386","reported_seq":9522,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.576438+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:15.576438+0000","last_peered":"2024-09-10T20:27:15.576438+0000","last_clean":"2024-09-10T20:27:15.576438+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:15.576438+0000","last_undegraded":"2024-09-10T20:27:15.576438+0000","last_fullsized":"2024-09-10T20:27:15.576438+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23478272,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5827,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9264","reported_seq":9471,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.575640+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:14.575640+0000","last_peered":"2024-09-10T20:27:14.575640+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:14.575640+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9264,"log_dups_size":0,"ondisk_log_size":9264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23558760,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9263,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9504","reported_seq":13676,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.760269+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:15.760269+0000","last_peered":"2024-09-10T20:27:15.760269+0000","last_clean":"2024-09-10T20:27:15.760269+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:15.760269+0000","last_undegraded":"2024-09-10T20:27:15.760269+0000","last_fullsized":"2024-09-10T20:27:15.760269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23773184,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9313","reported_seq":9636,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.829807+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:12.829807+0000","last_peered":"2024-09-10T20:27:12.829807+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:12.829807+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9313,"log_dups_size":0,"ondisk_log_size":9313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23393440,"num_objects":2856,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9311,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9273","reported_seq":9499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.790204+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:12.790204+0000","last_peered":"2024-09-10T20:27:12.790204+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:12.790204+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9273,"log_dups_size":0,"ondisk_log_size":9273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23142400,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9273,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9223","reported_seq":9430,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:11.815125+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:11.815125+0000","last_peered":"2024-09-10T20:27:11.815125+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:11.815125+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9223,"log_dups_size":0,"ondisk_log_size":9223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23319712,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9221,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9357","reported_seq":9529,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:11.966995+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:11.966995+0000","last_peered":"2024-09-10T20:27:11.966995+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:11.966995+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24048800,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9150","reported_seq":13126,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:12.957329+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:12.957329+0000","last_peered":"2024-09-10T20:27:12.957329+0000","last_clean":"2024-09-10T20:27:12.957329+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:12.957329+0000","last_undegraded":"2024-09-10T20:27:12.957329+0000","last_fullsized":"2024-09-10T20:27:12.957329+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9150,"log_dups_size":0,"ondisk_log_size":9150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23353936,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9190","reported_seq":13165,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:14.569606+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:14.569606+0000","last_peered":"2024-09-10T20:27:14.569606+0000","last_clean":"2024-09-10T20:27:14.569606+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:14.569606+0000","last_undegraded":"2024-09-10T20:27:14.569606+0000","last_fullsized":"2024-09-10T20:27:14.569606+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":2952,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1828,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9206","reported_seq":9483,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.789029+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:12.789029+0000","last_peered":"2024-09-10T20:27:12.789029+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:12.789029+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9206,"log_dups_size":0,"ondisk_log_size":9206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23809776,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9304","reported_seq":9605,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.825322+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:12.825322+0000","last_peered":"2024-09-10T20:27:12.825322+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:12.825322+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23435856,"num_objects":2861,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9037","reported_seq":9223,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.513358+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:14.513358+0000","last_peered":"2024-09-10T20:27:14.513358+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:14.513358+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9037,"log_dups_size":0,"ondisk_log_size":9037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22886992,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":891}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9249","reported_seq":9798,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:12.954060+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:12.954060+0000","last_peered":"2024-09-10T20:27:12.954060+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:12.954060+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303328,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9483","reported_seq":9692,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.570791+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:14.570791+0000","last_peered":"2024-09-10T20:27:14.570791+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:14.570791+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24294560,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9481,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9284","reported_seq":13383,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.714168+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:15.714168+0000","last_peered":"2024-09-10T20:27:15.714168+0000","last_clean":"2024-09-10T20:27:15.714168+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:15.714168+0000","last_undegraded":"2024-09-10T20:27:15.714168+0000","last_fullsized":"2024-09-10T20:27:15.714168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23257088,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376897384,"num_objects":46012,"num_object_clones":0,"num_object_copies":184048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46012,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119206,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":753758208,"data_stored":753758208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148457,"ondisk_log_size":148457,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690680,"kb_used_data":185572,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93681160,"statfs":{"total":96636764160,"available":95929507840,"internally_reserved":0,"allocated":190025728,"data_stored":189803212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,6,5,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693284,"kb_used_data":184520,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93678556,"statfs":{"total":96636764160,"available":95926841344,"internally_reserved":0,"allocated":188948480,"data_stored":188725964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616148,"kb_used_data":183228,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93755692,"statfs":{"total":96636764160,"available":96005828608,"internally_reserved":0,"allocated":187625472,"data_stored":187402956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691192,"kb_used_data":186144,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93680648,"statfs":{"total":96636764160,"available":95928983552,"internally_reserved":0,"allocated":190611456,"data_stored":190388940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190124032,"data_stored":190124032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186662912,"data_stored":186662912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187682816,"data_stored":187682816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":189288448,"data_stored":189288448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9234", 'reported_seq': 9519, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:12.979599+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:12.979599+0000', 'last_peered': '2024-09-10T20:27:12.979599+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:12.979599+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9234, 'log_dups_size': 0, 'ondisk_log_size': 9234, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23658496, 'num_objects': 2888, 'num_object_clones': 0, 'num_object_copies': 11552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9234, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1033}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9264", 'reported_seq': 9471, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:14.575640+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:14.575640+0000', 'last_peered': '2024-09-10T20:27:14.575640+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:14.575640+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9264, 'log_dups_size': 0, 'ondisk_log_size': 9264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23558760, 'num_objects': 2877, 'num_object_clones': 0, 'num_object_copies': 11508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2877, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9263, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 969}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9313", 'reported_seq': 9636, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:12.829807+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:12.829807+0000', 'last_peered': '2024-09-10T20:27:12.829807+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:12.829807+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9313, 'log_dups_size': 0, 'ondisk_log_size': 9313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23393440, 'num_objects': 2856, 'num_object_clones': 0, 'num_object_copies': 11424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9311, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1001}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9273", 'reported_seq': 9499, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:12.790204+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:12.790204+0000', 'last_peered': '2024-09-10T20:27:12.790204+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:12.790204+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9273, 'log_dups_size': 0, 'ondisk_log_size': 9273, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23142400, 'num_objects': 2825, 'num_object_clones': 0, 'num_object_copies': 11300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9273, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:17.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9223", 'reported_seq': 9430, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:11.815125+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:11.815125+0000', 'last_peered': '2024-09-10T20:27:11.815125+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:11.815125+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9223, 'log_dups_size': 0, 'ondisk_log_size': 9223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23319712, 'num_objects': 2847, 'num_object_clones': 0, 'num_object_copies': 11388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9221, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 954}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9357", 'reported_seq': 9529, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:11.966995+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:11.966995+0000', 'last_peered': '2024-09-10T20:27:11.966995+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:11.966995+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9357, 'log_dups_size': 0, 'ondisk_log_size': 9357, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24048800, 'num_objects': 2936, 'num_object_clones': 0, 'num_object_copies': 11744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9355, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9206", 'reported_seq': 9483, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:12.789029+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:12.789029+0000', 'last_peered': '2024-09-10T20:27:12.789029+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:12.789029+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9206, 'log_dups_size': 0, 'ondisk_log_size': 9206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23809776, 'num_objects': 2907, 'num_object_clones': 0, 'num_object_copies': 11628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9203, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1051}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9304", 'reported_seq': 9605, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:12.825322+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:12.825322+0000', 'last_peered': '2024-09-10T20:27:12.825322+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:12.825322+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9304, 'log_dups_size': 0, 'ondisk_log_size': 9304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23435856, 'num_objects': 2861, 'num_object_clones': 0, 'num_object_copies': 11444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9303, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1001}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9037", 'reported_seq': 9223, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:14.513358+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:14.513358+0000', 'last_peered': '2024-09-10T20:27:14.513358+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:14.513358+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9037, 'log_dups_size': 0, 'ondisk_log_size': 9037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22886992, 'num_objects': 2794, 'num_object_clones': 0, 'num_object_copies': 11176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9036, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 891}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:17.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9249", 'reported_seq': 9798, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:12.954060+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:12.954060+0000', 'last_peered': '2024-09-10T20:27:12.954060+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:12.954060+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9249, 'log_dups_size': 0, 'ondisk_log_size': 9249, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23303328, 'num_objects': 2845, 'num_object_clones': 0, 'num_object_copies': 11380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1739, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9247, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 240, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:17.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:17.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9483", 'reported_seq': 9692, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:14.570791+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:14.570791+0000', 'last_peered': '2024-09-10T20:27:14.570791+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:14.570791+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9483, 'log_dups_size': 0, 'ondisk_log_size': 9483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24294560, 'num_objects': 2966, 'num_object_clones': 0, 'num_object_copies': 11864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9481, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:17.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:17.950 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:17.950 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20648/184048 objects degraded (11.219%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":526,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":46012,"data_bytes":376897384,"bytes_used":2755895296,"bytes_avail":383791161344,"bytes_total":386547056640,"degraded_objects":20648,"degraded_total":184048,"degraded_ratio":0.11218812483699905},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:27:17.985 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:27:17.986 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:18.614 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:18.615 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:18.649 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-10T20:27:18.049230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371842920,"num_objects":45395,"num_object_clones":0,"num_object_copies":181580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119823,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149074,"ondisk_log_size":149074,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690624,"kb_used_data":730464,"kb_used_omap":14,"kb_used_meta":1960049,"kb_avail":374796736,"statfs":{"total":386547056640,"available":383791857664,"internally_reserved":0,"allocated":747995136,"data_stored":747105072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2007090742},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,4,16,18,21,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23352480,"num_objects":-2851,"num_object_clones":0,"num_object_copies":-11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6276,"num_write_kb":-48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1836,"num_bytes_recovered":-4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004202"},"pg_stats":[{"pgid":"1.f","version":"42'9307","reported_seq":9593,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.986128+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:17.986128+0000","last_peered":"2024-09-10T20:27:17.986128+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:17.986128+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23060480,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9386","reported_seq":9522,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.576438+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:15.576438+0000","last_peered":"2024-09-10T20:27:15.576438+0000","last_clean":"2024-09-10T20:27:15.576438+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:15.576438+0000","last_undegraded":"2024-09-10T20:27:15.576438+0000","last_fullsized":"2024-09-10T20:27:15.576438+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23478272,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5827,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9264","reported_seq":9471,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.575640+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:14.575640+0000","last_peered":"2024-09-10T20:27:14.575640+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:14.575640+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9264,"log_dups_size":0,"ondisk_log_size":9264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23558760,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9263,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9504","reported_seq":13676,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.760269+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:15.760269+0000","last_peered":"2024-09-10T20:27:15.760269+0000","last_clean":"2024-09-10T20:27:15.760269+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:15.760269+0000","last_undegraded":"2024-09-10T20:27:15.760269+0000","last_fullsized":"2024-09-10T20:27:15.760269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23773184,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9382","reported_seq":9705,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.987455+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:17.987455+0000","last_peered":"2024-09-10T20:27:17.987455+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:17.987455+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22828192,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9337","reported_seq":9563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.832336+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:17.832336+0000","last_peered":"2024-09-10T20:27:17.832336+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:17.832336+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9337,"log_dups_size":0,"ondisk_log_size":9337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22618112,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9292","reported_seq":9499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:16.859848+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:16.859848+0000","last_peered":"2024-09-10T20:27:16.859848+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:16.859848+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22754464,"num_objects":2778,"num_object_clones":0,"num_object_copies":11112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":885}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9426","reported_seq":9598,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:16.812215+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:16.812215+0000","last_peered":"2024-09-10T20:27:16.812215+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:16.812215+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23483552,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9224","reported_seq":13199,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:17.888753+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:17.888753+0000","last_peered":"2024-09-10T20:27:17.888753+0000","last_clean":"2024-09-10T20:27:17.888753+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:17.888753+0000","last_undegraded":"2024-09-10T20:27:17.888753+0000","last_fullsized":"2024-09-10T20:27:17.888753+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22747728,"num_objects":2777,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9190","reported_seq":13165,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:14.569606+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:14.569606+0000","last_peered":"2024-09-10T20:27:14.569606+0000","last_clean":"2024-09-10T20:27:14.569606+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:14.569606+0000","last_undegraded":"2024-09-10T20:27:14.569606+0000","last_fullsized":"2024-09-10T20:27:14.569606+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":2952,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1828,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9279","reported_seq":9556,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.834577+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:17.834577+0000","last_peered":"2024-09-10T20:27:17.834577+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:17.834577+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23211760,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9365","reported_seq":9667,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.920707+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:17.920707+0000","last_peered":"2024-09-10T20:27:17.920707+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:17.920707+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22936144,"num_objects":2800,"num_object_clones":0,"num_object_copies":11200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9037","reported_seq":9223,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.513358+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:14.513358+0000","last_peered":"2024-09-10T20:27:14.513358+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:14.513358+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9037,"log_dups_size":0,"ondisk_log_size":9037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22886992,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":891}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9314","reported_seq":9960,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.933952+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:17.933952+0000","last_peered":"2024-09-10T20:27:17.933952+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:17.933952+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22770848,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9483","reported_seq":9692,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:14.570791+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:14.570791+0000","last_peered":"2024-09-10T20:27:14.570791+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:14.570791+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24294560,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9481,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9284","reported_seq":13383,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:15.714168+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:15.714168+0000","last_peered":"2024-09-10T20:27:15.714168+0000","last_clean":"2024-09-10T20:27:15.714168+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:15.714168+0000","last_undegraded":"2024-09-10T20:27:15.714168+0000","last_fullsized":"2024-09-10T20:27:15.714168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23257088,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":371842920,"num_objects":45395,"num_object_clones":0,"num_object_copies":181580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119823,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":744595456,"data_stored":744595456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149074,"ondisk_log_size":149074,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690404,"kb_used_data":181132,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93681436,"statfs":{"total":96636764160,"available":95929790464,"internally_reserved":0,"allocated":185479168,"data_stored":185256652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,8,12,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693284,"kb_used_data":184520,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93678556,"statfs":{"total":96636764160,"available":95926841344,"internally_reserved":0,"allocated":188948480,"data_stored":188725964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616148,"kb_used_data":183228,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93755692,"statfs":{"total":96636764160,"available":96005828608,"internally_reserved":0,"allocated":187625472,"data_stored":187402956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690788,"kb_used_data":181584,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93681052,"statfs":{"total":96636764160,"available":95929397248,"internally_reserved":0,"allocated":185942016,"data_stored":185719500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,0,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185532416,"data_stored":185532416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186662912,"data_stored":186662912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187682816,"data_stored":187682816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184717312,"data_stored":184717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:21.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:22.213 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:22.213 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:22.251 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-10T20:27:22.050885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367886184,"num_objects":44912,"num_object_clones":0,"num_object_copies":179648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120306,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149557,"ondisk_log_size":149557,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673684,"kb_used_data":721848,"kb_used_omap":14,"kb_used_meta":1951729,"kb_avail":374813676,"statfs":{"total":386547056640,"available":383809204224,"internally_reserved":0,"allocated":739172352,"data_stored":738282288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1998571062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,1,17,20,18,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19887264,"num_objects":-2428,"num_object_clones":0,"num_object_copies":-9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6699,"num_write_kb":-48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1877,"num_bytes_recovered":-4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004807"},"pg_stats":[{"pgid":"1.f","version":"42'9307","reported_seq":9593,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.986128+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:17.986128+0000","last_peered":"2024-09-10T20:27:17.986128+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:17.986128+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23060480,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9453","reported_seq":9589,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.632809+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:20.632809+0000","last_peered":"2024-09-10T20:27:20.632809+0000","last_clean":"2024-09-10T20:27:20.632809+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:20.632809+0000","last_undegraded":"2024-09-10T20:27:20.632809+0000","last_fullsized":"2024-09-10T20:27:20.632809+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22929408,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9326","reported_seq":9530,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.637965+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:19.637965+0000","last_peered":"2024-09-10T20:27:19.637965+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:19.637965+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9326,"log_dups_size":0,"ondisk_log_size":9326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23050856,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9325,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9584","reported_seq":13758,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.827329+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:20.827329+0000","last_peered":"2024-09-10T20:27:20.827329+0000","last_clean":"2024-09-10T20:27:20.827329+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:20.827329+0000","last_undegraded":"2024-09-10T20:27:20.827329+0000","last_fullsized":"2024-09-10T20:27:20.827329+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9382","reported_seq":9705,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.987455+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:17.987455+0000","last_peered":"2024-09-10T20:27:17.987455+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:17.987455+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22828192,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9337","reported_seq":9563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.832336+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:17.832336+0000","last_peered":"2024-09-10T20:27:17.832336+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:17.832336+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9337,"log_dups_size":0,"ondisk_log_size":9337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22618112,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9292","reported_seq":9499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:16.859848+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:16.859848+0000","last_peered":"2024-09-10T20:27:16.859848+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:16.859848+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22754464,"num_objects":2778,"num_object_clones":0,"num_object_copies":11112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":885}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9426","reported_seq":9598,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:16.812215+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:16.812215+0000","last_peered":"2024-09-10T20:27:16.812215+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:16.812215+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23483552,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9224","reported_seq":13199,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:17.888753+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:17.888753+0000","last_peered":"2024-09-10T20:27:17.888753+0000","last_clean":"2024-09-10T20:27:17.888753+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:17.888753+0000","last_undegraded":"2024-09-10T20:27:17.888753+0000","last_fullsized":"2024-09-10T20:27:17.888753+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22747728,"num_objects":2777,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9259","reported_seq":13234,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:19.474155+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:19.474155+0000","last_peered":"2024-09-10T20:27:19.474155+0000","last_clean":"2024-09-10T20:27:19.474155+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:19.474155+0000","last_undegraded":"2024-09-10T20:27:19.474155+0000","last_fullsized":"2024-09-10T20:27:19.474155+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23617536,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1897,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9279","reported_seq":9556,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.834577+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:17.834577+0000","last_peered":"2024-09-10T20:27:17.834577+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:17.834577+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23211760,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9365","reported_seq":9667,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.920707+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:17.920707+0000","last_peered":"2024-09-10T20:27:17.920707+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:17.920707+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22936144,"num_objects":2800,"num_object_clones":0,"num_object_copies":11200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9100","reported_seq":9287,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.492115+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:19.492115+0000","last_peered":"2024-09-10T20:27:19.492115+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:19.492115+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9100,"log_dups_size":0,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22370896,"num_objects":2731,"num_object_clones":0,"num_object_copies":10924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9314","reported_seq":9960,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.933952+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:17.933952+0000","last_peered":"2024-09-10T20:27:17.933952+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:17.933952+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22770848,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9559","reported_seq":9768,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.563845+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:19.563845+0000","last_peered":"2024-09-10T20:27:19.563845+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:19.563845+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23671968,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9350","reported_seq":13450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.827569+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:20.827569+0000","last_peered":"2024-09-10T20:27:20.827569+0000","last_clean":"2024-09-10T20:27:20.827569+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:20.827569+0000","last_undegraded":"2024-09-10T20:27:20.827569+0000","last_fullsized":"2024-09-10T20:27:20.827569+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22716416,"num_objects":2773,"num_object_clones":0,"num_object_copies":11092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367886184,"num_objects":44912,"num_object_clones":0,"num_object_copies":179648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120306,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":735526912,"data_stored":735526912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149557,"ondisk_log_size":149557,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690404,"kb_used_data":181132,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93681436,"statfs":{"total":96636764160,"available":95929790464,"internally_reserved":0,"allocated":185479168,"data_stored":185256652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,8,12,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676368,"kb_used_data":180080,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93695472,"statfs":{"total":96636764160,"available":95944163328,"internally_reserved":0,"allocated":184401920,"data_stored":184179404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616124,"kb_used_data":179052,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93755716,"statfs":{"total":96636764160,"available":96005853184,"internally_reserved":0,"allocated":183349248,"data_stored":183126732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447540946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690788,"kb_used_data":181584,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93681052,"statfs":{"total":96636764160,"available":95929397248,"internally_reserved":0,"allocated":185942016,"data_stored":185719500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,0,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185532416,"data_stored":185532416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182124544,"data_stored":182124544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":183152640,"data_stored":183152640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184717312,"data_stored":184717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:22.252 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:22.966 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:22.966 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:22.986 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-10T20:27:22.050885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367886184,"num_objects":44912,"num_object_clones":0,"num_object_copies":179648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120306,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149557,"ondisk_log_size":149557,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673684,"kb_used_data":721848,"kb_used_omap":14,"kb_used_meta":1951729,"kb_avail":374813676,"statfs":{"total":386547056640,"available":383809204224,"internally_reserved":0,"allocated":739172352,"data_stored":738282288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1998571062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,1,17,20,18,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19887264,"num_objects":-2428,"num_object_clones":0,"num_object_copies":-9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6699,"num_write_kb":-48492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1877,"num_bytes_recovered":-4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004807"},"pg_stats":[{"pgid":"1.f","version":"42'9307","reported_seq":9593,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.986128+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:17.986128+0000","last_peered":"2024-09-10T20:27:17.986128+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:17.986128+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23060480,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9453","reported_seq":9589,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.632809+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:20.632809+0000","last_peered":"2024-09-10T20:27:20.632809+0000","last_clean":"2024-09-10T20:27:20.632809+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:20.632809+0000","last_undegraded":"2024-09-10T20:27:20.632809+0000","last_fullsized":"2024-09-10T20:27:20.632809+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22929408,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9326","reported_seq":9530,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.637965+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:19.637965+0000","last_peered":"2024-09-10T20:27:19.637965+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:19.637965+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9326,"log_dups_size":0,"ondisk_log_size":9326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23050856,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9325,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9584","reported_seq":13758,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.827329+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:20.827329+0000","last_peered":"2024-09-10T20:27:20.827329+0000","last_clean":"2024-09-10T20:27:20.827329+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:20.827329+0000","last_undegraded":"2024-09-10T20:27:20.827329+0000","last_fullsized":"2024-09-10T20:27:20.827329+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9382","reported_seq":9705,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.987455+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:17.987455+0000","last_peered":"2024-09-10T20:27:17.987455+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:17.987455+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22828192,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9337","reported_seq":9563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.832336+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:17.832336+0000","last_peered":"2024-09-10T20:27:17.832336+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:17.832336+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9337,"log_dups_size":0,"ondisk_log_size":9337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22618112,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9292","reported_seq":9499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:16.859848+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:16.859848+0000","last_peered":"2024-09-10T20:27:16.859848+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:16.859848+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22754464,"num_objects":2778,"num_object_clones":0,"num_object_copies":11112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":885}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9426","reported_seq":9598,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:16.812215+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:16.812215+0000","last_peered":"2024-09-10T20:27:16.812215+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:16.812215+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23483552,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9224","reported_seq":13199,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:17.888753+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:17.888753+0000","last_peered":"2024-09-10T20:27:17.888753+0000","last_clean":"2024-09-10T20:27:17.888753+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:17.888753+0000","last_undegraded":"2024-09-10T20:27:17.888753+0000","last_fullsized":"2024-09-10T20:27:17.888753+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22747728,"num_objects":2777,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9259","reported_seq":13234,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:19.474155+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:19.474155+0000","last_peered":"2024-09-10T20:27:19.474155+0000","last_clean":"2024-09-10T20:27:19.474155+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:19.474155+0000","last_undegraded":"2024-09-10T20:27:19.474155+0000","last_fullsized":"2024-09-10T20:27:19.474155+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23617536,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1897,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9279","reported_seq":9556,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.834577+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:17.834577+0000","last_peered":"2024-09-10T20:27:17.834577+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:17.834577+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23211760,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9365","reported_seq":9667,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.920707+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:17.920707+0000","last_peered":"2024-09-10T20:27:17.920707+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:17.920707+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22936144,"num_objects":2800,"num_object_clones":0,"num_object_copies":11200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9100","reported_seq":9287,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.492115+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:19.492115+0000","last_peered":"2024-09-10T20:27:19.492115+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:19.492115+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9100,"log_dups_size":0,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22370896,"num_objects":2731,"num_object_clones":0,"num_object_copies":10924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9314","reported_seq":9960,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:17.933952+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:17.933952+0000","last_peered":"2024-09-10T20:27:17.933952+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:17.933952+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22770848,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9559","reported_seq":9768,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.563845+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:19.563845+0000","last_peered":"2024-09-10T20:27:19.563845+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:19.563845+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23671968,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9350","reported_seq":13450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.827569+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:20.827569+0000","last_peered":"2024-09-10T20:27:20.827569+0000","last_clean":"2024-09-10T20:27:20.827569+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:20.827569+0000","last_undegraded":"2024-09-10T20:27:20.827569+0000","last_fullsized":"2024-09-10T20:27:20.827569+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22716416,"num_objects":2773,"num_object_clones":0,"num_object_copies":11092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367886184,"num_objects":44912,"num_object_clones":0,"num_object_copies":179648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120306,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":735526912,"data_stored":735526912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149557,"ondisk_log_size":149557,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690404,"kb_used_data":181132,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93681436,"statfs":{"total":96636764160,"available":95929790464,"internally_reserved":0,"allocated":185479168,"data_stored":185256652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,8,12,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676368,"kb_used_data":180080,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93695472,"statfs":{"total":96636764160,"available":95944163328,"internally_reserved":0,"allocated":184401920,"data_stored":184179404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616124,"kb_used_data":179052,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93755716,"statfs":{"total":96636764160,"available":96005853184,"internally_reserved":0,"allocated":183349248,"data_stored":183126732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447540946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690788,"kb_used_data":181584,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93681052,"statfs":{"total":96636764160,"available":95929397248,"internally_reserved":0,"allocated":185942016,"data_stored":185719500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,0,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185532416,"data_stored":185532416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182124544,"data_stored":182124544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":183152640,"data_stored":183152640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184717312,"data_stored":184717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:22.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:22.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9307", 'reported_seq': 9593, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:17.986128+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:17.986128+0000', 'last_peered': '2024-09-10T20:27:17.986128+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:17.986128+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9307, 'log_dups_size': 0, 'ondisk_log_size': 9307, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23060480, 'num_objects': 2815, 'num_object_clones': 0, 'num_object_copies': 11260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9307, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 960}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9326", 'reported_seq': 9530, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:19.637965+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:19.637965+0000', 'last_peered': '2024-09-10T20:27:19.637965+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:19.637965+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9326, 'log_dups_size': 0, 'ondisk_log_size': 9326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23050856, 'num_objects': 2815, 'num_object_clones': 0, 'num_object_copies': 11260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2815, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9325, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9382", 'reported_seq': 9705, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:17.987455+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:17.987455+0000', 'last_peered': '2024-09-10T20:27:17.987455+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:17.987455+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9382, 'log_dups_size': 0, 'ondisk_log_size': 9382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22828192, 'num_objects': 2787, 'num_object_clones': 0, 'num_object_copies': 11148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9380, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 932}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9337", 'reported_seq': 9563, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:17.832336+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:17.832336+0000', 'last_peered': '2024-09-10T20:27:17.832336+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:17.832336+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9337, 'log_dups_size': 0, 'ondisk_log_size': 9337, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22618112, 'num_objects': 2761, 'num_object_clones': 0, 'num_object_copies': 11044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9337, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9292", 'reported_seq': 9499, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:16.859848+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:16.859848+0000', 'last_peered': '2024-09-10T20:27:16.859848+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:16.859848+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9292, 'log_dups_size': 0, 'ondisk_log_size': 9292, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22754464, 'num_objects': 2778, 'num_object_clones': 0, 'num_object_copies': 11112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9290, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 885}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:22.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9426", 'reported_seq': 9598, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:16.812215+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:16.812215+0000', 'last_peered': '2024-09-10T20:27:16.812215+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:16.812215+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9426, 'log_dups_size': 0, 'ondisk_log_size': 9426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23483552, 'num_objects': 2867, 'num_object_clones': 0, 'num_object_copies': 11468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9424, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 867}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9279", 'reported_seq': 9556, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:17.834577+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:17.834577+0000', 'last_peered': '2024-09-10T20:27:17.834577+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:17.834577+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9279, 'log_dups_size': 0, 'ondisk_log_size': 9279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23211760, 'num_objects': 2834, 'num_object_clones': 0, 'num_object_copies': 11336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9276, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 978}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9365", 'reported_seq': 9667, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:17.920707+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:17.920707+0000', 'last_peered': '2024-09-10T20:27:17.920707+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:17.920707+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9365, 'log_dups_size': 0, 'ondisk_log_size': 9365, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22936144, 'num_objects': 2800, 'num_object_clones': 0, 'num_object_copies': 11200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9364, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9100", 'reported_seq': 9287, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:19.492115+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:19.492115+0000', 'last_peered': '2024-09-10T20:27:19.492115+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:19.492115+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9100, 'log_dups_size': 0, 'ondisk_log_size': 9100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22370896, 'num_objects': 2731, 'num_object_clones': 0, 'num_object_copies': 10924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9099, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9314", 'reported_seq': 9960, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:17.933952+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:17.933952+0000', 'last_peered': '2024-09-10T20:27:17.933952+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:17.933952+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9314, 'log_dups_size': 0, 'ondisk_log_size': 9314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22770848, 'num_objects': 2780, 'num_object_clones': 0, 'num_object_copies': 11120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1691, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9312, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:22.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9559", 'reported_seq': 9768, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:19.563845+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:19.563845+0000', 'last_peered': '2024-09-10T20:27:19.563845+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:19.563845+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9559, 'log_dups_size': 0, 'ondisk_log_size': 9559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23671968, 'num_objects': 2890, 'num_object_clones': 0, 'num_object_copies': 11560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9557, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 982}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:22.990 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:23.770 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:23.771 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20600/179648 objects degraded (11.467%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":532,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44912,"data_bytes":367886184,"bytes_used":2737852416,"bytes_avail":383809204224,"bytes_total":386547056640,"degraded_objects":20600,"degraded_total":179648,"degraded_ratio":0.11466868542928393},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:27:23.802 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:27:23.802 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:24.277 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:24.277 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:24.318 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-10T20:27:24.051690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":362864488,"num_objects":44299,"num_object_clones":0,"num_object_copies":177196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120919,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150170,"ondisk_log_size":150170,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2656528,"kb_used_data":713024,"kb_used_omap":14,"kb_used_meta":1943409,"kb_avail":374830832,"statfs":{"total":386547056640,"available":383826771968,"internally_reserved":0,"allocated":730136576,"data_stored":729246512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1990051382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,9,4,9,13,21,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22265856,"num_objects":-2718,"num_object_clones":0,"num_object_copies":-10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004995"},"pg_stats":[{"pgid":"1.f","version":"42'9378","reported_seq":9664,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.765431+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:22.765431+0000","last_peered":"2024-09-10T20:27:22.765431+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:22.765431+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9378,"log_dups_size":0,"ondisk_log_size":9378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9378,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9453","reported_seq":9589,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.632809+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:20.632809+0000","last_peered":"2024-09-10T20:27:20.632809+0000","last_clean":"2024-09-10T20:27:20.632809+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:20.632809+0000","last_undegraded":"2024-09-10T20:27:20.632809+0000","last_fullsized":"2024-09-10T20:27:20.632809+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22929408,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9326","reported_seq":9530,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.637965+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:19.637965+0000","last_peered":"2024-09-10T20:27:19.637965+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:19.637965+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9326,"log_dups_size":0,"ondisk_log_size":9326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23050856,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9325,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9584","reported_seq":13758,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.827329+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:20.827329+0000","last_peered":"2024-09-10T20:27:20.827329+0000","last_clean":"2024-09-10T20:27:20.827329+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:20.827329+0000","last_undegraded":"2024-09-10T20:27:20.827329+0000","last_fullsized":"2024-09-10T20:27:20.827329+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9446","reported_seq":9770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.749774+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:22.749774+0000","last_peered":"2024-09-10T20:27:22.749774+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:22.749774+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9446,"log_dups_size":0,"ondisk_log_size":9446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22303904,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9444,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9407","reported_seq":9632,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.932992+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:22.932992+0000","last_peered":"2024-09-10T20:27:22.932992+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:22.932992+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22044672,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":834}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9358","reported_seq":9564,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.073640+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:22.073640+0000","last_peered":"2024-09-10T20:27:22.073640+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:22.073640+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9358,"log_dups_size":0,"ondisk_log_size":9358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22213792,"num_objects":2712,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9356,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9499","reported_seq":9671,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:21.956445+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:21.956445+0000","last_peered":"2024-09-10T20:27:21.956445+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:21.956445+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885536,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9284","reported_seq":13260,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:22.929997+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:22.929997+0000","last_peered":"2024-09-10T20:27:22.929997+0000","last_clean":"2024-09-10T20:27:22.929997+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:22.929997+0000","last_undegraded":"2024-09-10T20:27:22.929997+0000","last_fullsized":"2024-09-10T20:27:22.929997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22256208,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9259","reported_seq":13234,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:19.474155+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:19.474155+0000","last_peered":"2024-09-10T20:27:19.474155+0000","last_clean":"2024-09-10T20:27:19.474155+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:19.474155+0000","last_undegraded":"2024-09-10T20:27:19.474155+0000","last_fullsized":"2024-09-10T20:27:19.474155+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23617536,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1897,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9348","reported_seq":9625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.983526+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:22.983526+0000","last_peered":"2024-09-10T20:27:22.983526+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:22.983526+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22646512,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9438","reported_seq":9740,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.973460+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:22.973460+0000","last_peered":"2024-09-10T20:27:22.973460+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:22.973460+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22338128,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9100","reported_seq":9287,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.492115+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:19.492115+0000","last_peered":"2024-09-10T20:27:19.492115+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:19.492115+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9100,"log_dups_size":0,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22370896,"num_objects":2731,"num_object_clones":0,"num_object_copies":10924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9381","reported_seq":10132,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.880687+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:22.880687+0000","last_peered":"2024-09-10T20:27:22.880687+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:22.880687+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22221984,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9559","reported_seq":9768,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:19.563845+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:19.563845+0000","last_peered":"2024-09-10T20:27:19.563845+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:19.563845+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23671968,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9350","reported_seq":13450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:20.827569+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:20.827569+0000","last_peered":"2024-09-10T20:27:20.827569+0000","last_clean":"2024-09-10T20:27:20.827569+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:20.827569+0000","last_undegraded":"2024-09-10T20:27:20.827569+0000","last_fullsized":"2024-09-10T20:27:20.827569+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22716416,"num_objects":2773,"num_object_clones":0,"num_object_copies":11092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":362864488,"num_objects":44299,"num_object_clones":0,"num_object_copies":177196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120919,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":726376448,"data_stored":726376448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150170,"ondisk_log_size":150170,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690132,"kb_used_data":176708,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93681708,"statfs":{"total":96636764160,"available":95930068992,"internally_reserved":0,"allocated":180948992,"data_stored":180726476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":525725392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,1,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676368,"kb_used_data":180080,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93695472,"statfs":{"total":96636764160,"available":95944163328,"internally_reserved":0,"allocated":184401920,"data_stored":184179404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616124,"kb_used_data":179052,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93755716,"statfs":{"total":96636764160,"available":96005853184,"internally_reserved":0,"allocated":183349248,"data_stored":183126732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447540946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215205,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673904,"kb_used_data":177184,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93697936,"statfs":{"total":96636764160,"available":95946686464,"internally_reserved":0,"allocated":181436416,"data_stored":181213900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180944896,"data_stored":180944896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182124544,"data_stored":182124544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":183152640,"data_stored":183152640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180154368,"data_stored":180154368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:27.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:28.572 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:28.572 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:28.595 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-10T20:27:28.052823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357842792,"num_objects":43686,"num_object_clones":0,"num_object_copies":174744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121532,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150783,"ondisk_log_size":150783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2638064,"kb_used_data":698712,"kb_used_omap":14,"kb_used_meta":1939249,"kb_avail":374849296,"statfs":{"total":386547056640,"available":383845679104,"internally_reserved":0,"allocated":715481088,"data_stored":714591024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1985791542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,8,7,4,13,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19054592,"num_objects":-2326,"num_object_clones":0,"num_object_copies":-9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004476"},"pg_stats":[{"pgid":"1.f","version":"42'9378","reported_seq":9664,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.765431+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:22.765431+0000","last_peered":"2024-09-10T20:27:22.765431+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:22.765431+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9378,"log_dups_size":0,"ondisk_log_size":9378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9378,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9515","reported_seq":9651,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.964311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:25.964311+0000","last_peered":"2024-09-10T20:27:25.964311+0000","last_clean":"2024-09-10T20:27:25.964311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:25.964311+0000","last_undegraded":"2024-09-10T20:27:25.964311+0000","last_fullsized":"2024-09-10T20:27:25.964311+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22421504,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5956,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9390","reported_seq":9597,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:24.576704+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:24.576704+0000","last_peered":"2024-09-10T20:27:24.576704+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:24.576704+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9390,"log_dups_size":0,"ondisk_log_size":9390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22526568,"num_objects":2751,"num_object_clones":0,"num_object_copies":11004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9389,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9662","reported_seq":13836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.910637+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:25.910637+0000","last_peered":"2024-09-10T20:27:25.910637+0000","last_clean":"2024-09-10T20:27:25.910637+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:25.910637+0000","last_undegraded":"2024-09-10T20:27:25.910637+0000","last_fullsized":"2024-09-10T20:27:25.910637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9662,"log_dups_size":0,"ondisk_log_size":9662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9446","reported_seq":9770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.749774+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:22.749774+0000","last_peered":"2024-09-10T20:27:22.749774+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:22.749774+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9446,"log_dups_size":0,"ondisk_log_size":9446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22303904,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9444,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9407","reported_seq":9632,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.932992+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:22.932992+0000","last_peered":"2024-09-10T20:27:22.932992+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:22.932992+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22044672,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":834}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9426","reported_seq":9633,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:26.757212+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:26.757212+0000","last_peered":"2024-09-10T20:27:26.757212+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:26.757212+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656736,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9580","reported_seq":9751,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:26.979295+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:26.979295+0000","last_peered":"2024-09-10T20:27:26.979295+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:26.979295+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22221984,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9284","reported_seq":13260,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:22.929997+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:22.929997+0000","last_peered":"2024-09-10T20:27:22.929997+0000","last_clean":"2024-09-10T20:27:22.929997+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:22.929997+0000","last_undegraded":"2024-09-10T20:27:22.929997+0000","last_fullsized":"2024-09-10T20:27:22.929997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22256208,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9320","reported_seq":13295,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:24.677300+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:24.677300+0000","last_peered":"2024-09-10T20:27:24.677300+0000","last_clean":"2024-09-10T20:27:24.677300+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:24.677300+0000","last_undegraded":"2024-09-10T20:27:24.677300+0000","last_fullsized":"2024-09-10T20:27:24.677300+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1958,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9348","reported_seq":9625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.983526+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:22.983526+0000","last_peered":"2024-09-10T20:27:22.983526+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:22.983526+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22646512,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9438","reported_seq":9740,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.973460+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:22.973460+0000","last_peered":"2024-09-10T20:27:22.973460+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:22.973460+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22338128,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9175","reported_seq":9362,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:24.743357+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:24.743357+0000","last_peered":"2024-09-10T20:27:24.743357+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:24.743357+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21756496,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9381","reported_seq":10132,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.880687+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:22.880687+0000","last_peered":"2024-09-10T20:27:22.880687+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:22.880687+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22221984,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9627","reported_seq":9836,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:24.220467+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:24.220467+0000","last_peered":"2024-09-10T20:27:24.220467+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:24.220467+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23114912,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9625,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":914}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9406","reported_seq":13506,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.746819+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:25.746819+0000","last_peered":"2024-09-10T20:27:25.746819+0000","last_clean":"2024-09-10T20:27:25.746819+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:25.746819+0000","last_undegraded":"2024-09-10T20:27:25.746819+0000","last_fullsized":"2024-09-10T20:27:25.746819+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22257664,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":357842792,"num_objects":43686,"num_object_clones":0,"num_object_copies":174744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121532,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":713289728,"data_stored":713289728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150783,"ondisk_log_size":150783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690132,"kb_used_data":176708,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93681708,"statfs":{"total":96636764160,"available":95930068992,"internally_reserved":0,"allocated":180948992,"data_stored":180726476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":525725392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,1,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675476,"kb_used_data":175044,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93696364,"statfs":{"total":96636764160,"available":95945076736,"internally_reserved":0,"allocated":179245056,"data_stored":179022540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598556,"kb_used_data":173956,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93773284,"statfs":{"total":96636764160,"available":96023842816,"internally_reserved":0,"allocated":178130944,"data_stored":177908428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":434761426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673900,"kb_used_data":173004,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93697940,"statfs":{"total":96636764160,"available":95946690560,"internally_reserved":0,"allocated":177156096,"data_stored":176933580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176590848,"data_stored":176590848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":177704960,"data_stored":177704960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178839552,"data_stored":178839552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180154368,"data_stored":180154368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:28.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:29.268 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:29.269 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:29.290 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-10T20:27:28.052823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357842792,"num_objects":43686,"num_object_clones":0,"num_object_copies":174744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121532,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150783,"ondisk_log_size":150783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2638064,"kb_used_data":698712,"kb_used_omap":14,"kb_used_meta":1939249,"kb_avail":374849296,"statfs":{"total":386547056640,"available":383845679104,"internally_reserved":0,"allocated":715481088,"data_stored":714591024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1985791542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,8,7,4,13,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19054592,"num_objects":-2326,"num_object_clones":0,"num_object_copies":-9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004476"},"pg_stats":[{"pgid":"1.f","version":"42'9378","reported_seq":9664,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.765431+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:22.765431+0000","last_peered":"2024-09-10T20:27:22.765431+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:22.765431+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9378,"log_dups_size":0,"ondisk_log_size":9378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9378,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9515","reported_seq":9651,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.964311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:25.964311+0000","last_peered":"2024-09-10T20:27:25.964311+0000","last_clean":"2024-09-10T20:27:25.964311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:25.964311+0000","last_undegraded":"2024-09-10T20:27:25.964311+0000","last_fullsized":"2024-09-10T20:27:25.964311+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22421504,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5956,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9390","reported_seq":9597,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:24.576704+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:24.576704+0000","last_peered":"2024-09-10T20:27:24.576704+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:24.576704+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9390,"log_dups_size":0,"ondisk_log_size":9390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22526568,"num_objects":2751,"num_object_clones":0,"num_object_copies":11004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9389,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9662","reported_seq":13836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.910637+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:25.910637+0000","last_peered":"2024-09-10T20:27:25.910637+0000","last_clean":"2024-09-10T20:27:25.910637+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:25.910637+0000","last_undegraded":"2024-09-10T20:27:25.910637+0000","last_fullsized":"2024-09-10T20:27:25.910637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9662,"log_dups_size":0,"ondisk_log_size":9662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9446","reported_seq":9770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.749774+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:22.749774+0000","last_peered":"2024-09-10T20:27:22.749774+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:22.749774+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9446,"log_dups_size":0,"ondisk_log_size":9446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22303904,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9444,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9407","reported_seq":9632,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.932992+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:22.932992+0000","last_peered":"2024-09-10T20:27:22.932992+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:22.932992+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22044672,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":834}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9426","reported_seq":9633,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:26.757212+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:26.757212+0000","last_peered":"2024-09-10T20:27:26.757212+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:26.757212+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656736,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9580","reported_seq":9751,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:26.979295+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:26.979295+0000","last_peered":"2024-09-10T20:27:26.979295+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:26.979295+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22221984,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9284","reported_seq":13260,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:22.929997+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:22.929997+0000","last_peered":"2024-09-10T20:27:22.929997+0000","last_clean":"2024-09-10T20:27:22.929997+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:22.929997+0000","last_undegraded":"2024-09-10T20:27:22.929997+0000","last_fullsized":"2024-09-10T20:27:22.929997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22256208,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9320","reported_seq":13295,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:24.677300+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:24.677300+0000","last_peered":"2024-09-10T20:27:24.677300+0000","last_clean":"2024-09-10T20:27:24.677300+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:24.677300+0000","last_undegraded":"2024-09-10T20:27:24.677300+0000","last_fullsized":"2024-09-10T20:27:24.677300+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1958,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9348","reported_seq":9625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.983526+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:22.983526+0000","last_peered":"2024-09-10T20:27:22.983526+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:22.983526+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22646512,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9438","reported_seq":9740,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.973460+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:22.973460+0000","last_peered":"2024-09-10T20:27:22.973460+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:22.973460+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22338128,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9175","reported_seq":9362,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:24.743357+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:24.743357+0000","last_peered":"2024-09-10T20:27:24.743357+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:24.743357+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21756496,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9381","reported_seq":10132,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:22.880687+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:22.880687+0000","last_peered":"2024-09-10T20:27:22.880687+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:22.880687+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22221984,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9627","reported_seq":9836,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:24.220467+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:24.220467+0000","last_peered":"2024-09-10T20:27:24.220467+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:24.220467+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23114912,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9625,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":914}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9406","reported_seq":13506,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.746819+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:25.746819+0000","last_peered":"2024-09-10T20:27:25.746819+0000","last_clean":"2024-09-10T20:27:25.746819+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:25.746819+0000","last_undegraded":"2024-09-10T20:27:25.746819+0000","last_fullsized":"2024-09-10T20:27:25.746819+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22257664,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":357842792,"num_objects":43686,"num_object_clones":0,"num_object_copies":174744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121532,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":713289728,"data_stored":713289728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150783,"ondisk_log_size":150783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690132,"kb_used_data":176708,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93681708,"statfs":{"total":96636764160,"available":95930068992,"internally_reserved":0,"allocated":180948992,"data_stored":180726476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":525725392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,1,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675476,"kb_used_data":175044,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93696364,"statfs":{"total":96636764160,"available":95945076736,"internally_reserved":0,"allocated":179245056,"data_stored":179022540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598556,"kb_used_data":173956,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93773284,"statfs":{"total":96636764160,"available":96023842816,"internally_reserved":0,"allocated":178130944,"data_stored":177908428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":434761426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673900,"kb_used_data":173004,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93697940,"statfs":{"total":96636764160,"available":95946690560,"internally_reserved":0,"allocated":177156096,"data_stored":176933580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176590848,"data_stored":176590848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":177704960,"data_stored":177704960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178839552,"data_stored":178839552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180154368,"data_stored":180154368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:29.291 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:29.291 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9378", 'reported_seq': 9664, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:22.765431+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:22.765431+0000', 'last_peered': '2024-09-10T20:27:22.765431+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:22.765431+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9378, 'log_dups_size': 0, 'ondisk_log_size': 9378, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22478848, 'num_objects': 2744, 'num_object_clones': 0, 'num_object_copies': 10976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9378, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 889}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:29.291 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:29.291 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9390", 'reported_seq': 9597, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:24.576704+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:24.576704+0000', 'last_peered': '2024-09-10T20:27:24.576704+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:24.576704+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9390, 'log_dups_size': 0, 'ondisk_log_size': 9390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22526568, 'num_objects': 2751, 'num_object_clones': 0, 'num_object_copies': 11004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2751, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9389, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 843}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:29.291 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9446", 'reported_seq': 9770, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:22.749774+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:22.749774+0000', 'last_peered': '2024-09-10T20:27:22.749774+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:22.749774+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9446, 'log_dups_size': 0, 'ondisk_log_size': 9446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22303904, 'num_objects': 2723, 'num_object_clones': 0, 'num_object_copies': 10892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9444, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 868}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9407", 'reported_seq': 9632, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:22.932992+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:22.932992+0000', 'last_peered': '2024-09-10T20:27:22.932992+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:22.932992+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9407, 'log_dups_size': 0, 'ondisk_log_size': 9407, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22044672, 'num_objects': 2691, 'num_object_clones': 0, 'num_object_copies': 10764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9407, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 834}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9426", 'reported_seq': 9633, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:26.757212+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:26.757212+0000', 'last_peered': '2024-09-10T20:27:26.757212+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:26.757212+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9426, 'log_dups_size': 0, 'ondisk_log_size': 9426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21656736, 'num_objects': 2644, 'num_object_clones': 0, 'num_object_copies': 10576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9424, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 751}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9580", 'reported_seq': 9751, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:26.979295+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:26.979295+0000', 'last_peered': '2024-09-10T20:27:26.979295+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:26.979295+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9580, 'log_dups_size': 0, 'ondisk_log_size': 9580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22221984, 'num_objects': 2713, 'num_object_clones': 0, 'num_object_copies': 10852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9578, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 713}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:29.292 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9348", 'reported_seq': 9625, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:22.983526+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:22.983526+0000', 'last_peered': '2024-09-10T20:27:22.983526+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:22.983526+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9348, 'log_dups_size': 0, 'ondisk_log_size': 9348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22646512, 'num_objects': 2765, 'num_object_clones': 0, 'num_object_copies': 11060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9345, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9438", 'reported_seq': 9740, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:22.973460+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:22.973460+0000', 'last_peered': '2024-09-10T20:27:22.973460+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:22.973460+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9438, 'log_dups_size': 0, 'ondisk_log_size': 9438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22338128, 'num_objects': 2727, 'num_object_clones': 0, 'num_object_copies': 10908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9437, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 867}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9175", 'reported_seq': 9362, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:24.743357+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:24.743357+0000', 'last_peered': '2024-09-10T20:27:24.743357+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:24.743357+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9175, 'log_dups_size': 0, 'ondisk_log_size': 9175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21756496, 'num_objects': 2656, 'num_object_clones': 0, 'num_object_copies': 10624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9174, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 753}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9381", 'reported_seq': 10132, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:22.880687+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:22.880687+0000', 'last_peered': '2024-09-10T20:27:22.880687+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:22.880687+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9381, 'log_dups_size': 0, 'ondisk_log_size': 9381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22221984, 'num_objects': 2713, 'num_object_clones': 0, 'num_object_copies': 10852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1637, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9379, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 342, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1076}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:29.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9627", 'reported_seq': 9836, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:24.220467+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:24.220467+0000', 'last_peered': '2024-09-10T20:27:24.220467+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:24.220467+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9627, 'log_dups_size': 0, 'ondisk_log_size': 9627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23114912, 'num_objects': 2822, 'num_object_clones': 0, 'num_object_copies': 11288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9625, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 914}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:29.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:30.024 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20546/174744 objects degraded (11.758%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":538,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43686,"data_bytes":357842792,"bytes_used":2701377536,"bytes_avail":383845679104,"bytes_total":386547056640,"degraded_objects":20546,"degraded_total":174744,"degraded_ratio":0.11757771368401776,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":193},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-10T20:24:57.989026+0000","services":{}},"progress_events":{}} 2024-09-10T20:27:30.053 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:30.054 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:30.658 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:30.658 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:30.703 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-10T20:27:30.053496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":351715176,"num_objects":42938,"num_object_clones":0,"num_object_copies":171752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122280,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151531,"ondisk_log_size":151531,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2620980,"kb_used_data":689924,"kb_used_omap":14,"kb_used_meta":1930929,"kb_avail":374866380,"statfs":{"total":386547056640,"available":383863173120,"internally_reserved":0,"allocated":706482176,"data_stored":705592112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1977271862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,14,14,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20127744,"num_objects":-2457,"num_object_clones":0,"num_object_copies":-9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2457,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004266"},"pg_stats":[{"pgid":"1.f","version":"42'9452","reported_seq":9737,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:27.959499+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:27.959499+0000","last_peered":"2024-09-10T20:27:27.959499+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:27.959499+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21872640,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9452,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9515","reported_seq":9651,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.964311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:25.964311+0000","last_peered":"2024-09-10T20:27:25.964311+0000","last_clean":"2024-09-10T20:27:25.964311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:25.964311+0000","last_undegraded":"2024-09-10T20:27:25.964311+0000","last_fullsized":"2024-09-10T20:27:25.964311+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22421504,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5956,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9450","reported_seq":9657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.441403+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:29.441403+0000","last_peered":"2024-09-10T20:27:29.441403+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:29.441403+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22035048,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9449,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9662","reported_seq":13836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.910637+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:25.910637+0000","last_peered":"2024-09-10T20:27:25.910637+0000","last_clean":"2024-09-10T20:27:25.910637+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:25.910637+0000","last_undegraded":"2024-09-10T20:27:25.910637+0000","last_fullsized":"2024-09-10T20:27:25.910637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9662,"log_dups_size":0,"ondisk_log_size":9662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9507","reported_seq":9831,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:27.923991+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:27.923991+0000","last_peered":"2024-09-10T20:27:27.923991+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:27.923991+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9507,"log_dups_size":0,"ondisk_log_size":9507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21804192,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":807}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9472","reported_seq":9698,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:27.912471+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:27.912471+0000","last_peered":"2024-09-10T20:27:27.912471+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:27.912471+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9472,"log_dups_size":0,"ondisk_log_size":9472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21512192,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9426","reported_seq":9633,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:26.757212+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:26.757212+0000","last_peered":"2024-09-10T20:27:26.757212+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:26.757212+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656736,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9580","reported_seq":9751,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:26.979295+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:26.979295+0000","last_peered":"2024-09-10T20:27:26.979295+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:26.979295+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22221984,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9340","reported_seq":13316,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:27.939631+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:27.939631+0000","last_peered":"2024-09-10T20:27:27.939631+0000","last_clean":"2024-09-10T20:27:27.939631+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:27.939631+0000","last_undegraded":"2024-09-10T20:27:27.939631+0000","last_fullsized":"2024-09-10T20:27:27.939631+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21797456,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9397","reported_seq":13372,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:29.661043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:29.661043+0000","last_peered":"2024-09-10T20:27:29.661043+0000","last_clean":"2024-09-10T20:27:29.661043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:29.661043+0000","last_undegraded":"2024-09-10T20:27:29.661043+0000","last_fullsized":"2024-09-10T20:27:29.661043+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":2745,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9421","reported_seq":9698,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:27.993058+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:27.993058+0000","last_peered":"2024-09-10T20:27:27.993058+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:27.993058+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9421,"log_dups_size":0,"ondisk_log_size":9421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22048496,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9506","reported_seq":9808,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:27.942943+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:27.942943+0000","last_peered":"2024-09-10T20:27:27.942943+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:27.942943+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21781072,"num_objects":2659,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9247","reported_seq":9434,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.486733+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:29.486733+0000","last_peered":"2024-09-10T20:27:29.486733+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:29.486733+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9247,"log_dups_size":0,"ondisk_log_size":9247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21166672,"num_objects":2584,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9246,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9449","reported_seq":10292,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:28.051811+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:28.051811+0000","last_peered":"2024-09-10T20:27:28.051811+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:28.051811+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21664928,"num_objects":2645,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":387,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1053}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9701","reported_seq":9910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.676694+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:29.676694+0000","last_peered":"2024-09-10T20:27:29.676694+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:29.676694+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22508704,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":840}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9406","reported_seq":13506,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:25.746819+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:25.746819+0000","last_peered":"2024-09-10T20:27:25.746819+0000","last_clean":"2024-09-10T20:27:25.746819+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:25.746819+0000","last_undegraded":"2024-09-10T20:27:25.746819+0000","last_fullsized":"2024-09-10T20:27:25.746819+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22257664,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":351715176,"num_objects":42938,"num_object_clones":0,"num_object_copies":171752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122280,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":704311296,"data_stored":704311296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151531,"ondisk_log_size":151531,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117093,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673332,"kb_used_data":172364,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93698508,"statfs":{"total":96636764160,"available":95947272192,"internally_reserved":0,"allocated":176500736,"data_stored":176278220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,4,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675192,"kb_used_data":170600,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93696648,"statfs":{"total":96636764160,"available":95945367552,"internally_reserved":0,"allocated":174694400,"data_stored":174471884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598556,"kb_used_data":173956,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93773284,"statfs":{"total":96636764160,"available":96023842816,"internally_reserved":0,"allocated":178130944,"data_stored":177908428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":434761426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673900,"kb_used_data":173004,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93697940,"statfs":{"total":96636764160,"available":95946690560,"internally_reserved":0,"allocated":177156096,"data_stored":176933580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176590848,"data_stored":176590848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":177704960,"data_stored":177704960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174272512,"data_stored":174272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":175742976,"data_stored":175742976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:33.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:34.317 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:34.318 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:34.337 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-10T20:27:34.055248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344956776,"num_objects":42113,"num_object_clones":0,"num_object_copies":168452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123105,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152356,"ondisk_log_size":152356,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2620360,"kb_used_data":676856,"kb_used_omap":14,"kb_used_meta":1943409,"kb_avail":374867000,"statfs":{"total":386547056640,"available":383863808000,"internally_reserved":0,"allocated":693100544,"data_stored":692210480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1990051382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,16,18,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22929408,"num_objects":-2799,"num_object_clones":0,"num_object_copies":-11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004363"},"pg_stats":[{"pgid":"1.f","version":"42'9510","reported_seq":9796,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.095002+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:33.095002+0000","last_peered":"2024-09-10T20:27:33.095002+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:33.095002+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9510,"log_dups_size":0,"ondisk_log_size":9510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397504,"num_objects":2612,"num_object_clones":0,"num_object_copies":10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9576","reported_seq":9712,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.771587+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:30.771587+0000","last_peered":"2024-09-10T20:27:30.771587+0000","last_clean":"2024-09-10T20:27:30.771587+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:30.771587+0000","last_undegraded":"2024-09-10T20:27:30.771587+0000","last_fullsized":"2024-09-10T20:27:30.771587+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21921792,"num_objects":2676,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9450","reported_seq":9657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.441403+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:29.441403+0000","last_peered":"2024-09-10T20:27:29.441403+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:29.441403+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22035048,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9449,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9729","reported_seq":13903,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.830573+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:30.830573+0000","last_peered":"2024-09-10T20:27:30.830573+0000","last_clean":"2024-09-10T20:27:30.830573+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:30.830573+0000","last_undegraded":"2024-09-10T20:27:30.830573+0000","last_fullsized":"2024-09-10T20:27:30.830573+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21929984,"num_objects":2677,"num_object_clones":0,"num_object_copies":10708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9570","reported_seq":9894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.129001+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:33.129001+0000","last_peered":"2024-09-10T20:27:33.129001+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:33.129001+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21288096,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9547","reported_seq":9772,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.037208+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:33.037208+0000","last_peered":"2024-09-10T20:27:33.037208+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:33.037208+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20897792,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9495","reported_seq":9702,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.092164+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:32.092164+0000","last_peered":"2024-09-10T20:27:32.092164+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:32.092164+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091488,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9659","reported_seq":9831,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.041691+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:32.041691+0000","last_peered":"2024-09-10T20:27:32.041691+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:32.041691+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9659,"log_dups_size":0,"ondisk_log_size":9659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574816,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9402","reported_seq":13378,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:33.055687+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:33.055687+0000","last_peered":"2024-09-10T20:27:33.055687+0000","last_clean":"2024-09-10T20:27:33.055687+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:33.055687+0000","last_undegraded":"2024-09-10T20:27:33.055687+0000","last_fullsized":"2024-09-10T20:27:33.055687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21289552,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9397","reported_seq":13372,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:29.661043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:29.661043+0000","last_peered":"2024-09-10T20:27:29.661043+0000","last_clean":"2024-09-10T20:27:29.661043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:29.661043+0000","last_undegraded":"2024-09-10T20:27:29.661043+0000","last_fullsized":"2024-09-10T20:27:29.661043+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":2745,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9495","reported_seq":9770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.147643+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:33.147643+0000","last_peered":"2024-09-10T20:27:33.147643+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:33.147643+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21442288,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9492,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":762}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9573","reported_seq":9875,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.953824+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:32.953824+0000","last_peered":"2024-09-10T20:27:32.953824+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:32.953824+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21232208,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":732}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9247","reported_seq":9434,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.486733+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:29.486733+0000","last_peered":"2024-09-10T20:27:29.486733+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:29.486733+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9247,"log_dups_size":0,"ondisk_log_size":9247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21166672,"num_objects":2584,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9246,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9528","reported_seq":10456,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.086415+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:33.086415+0000","last_peered":"2024-09-10T20:27:33.086415+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:33.086415+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9528,"log_dups_size":0,"ondisk_log_size":9528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21017760,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1016}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9701","reported_seq":9910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.676694+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:29.676694+0000","last_peered":"2024-09-10T20:27:29.676694+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:29.676694+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22508704,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":840}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9477","reported_seq":13577,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.768788+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:30.768788+0000","last_peered":"2024-09-10T20:27:30.768788+0000","last_clean":"2024-09-10T20:27:30.768788+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:30.768788+0000","last_undegraded":"2024-09-10T20:27:30.768788+0000","last_fullsized":"2024-09-10T20:27:30.768788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21676032,"num_objects":2646,"num_object_clones":0,"num_object_copies":10584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":344956776,"num_objects":42113,"num_object_clones":0,"num_object_copies":168452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123105,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":690741248,"data_stored":690741248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152356,"ondisk_log_size":152356,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673116,"kb_used_data":168008,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93698724,"statfs":{"total":96636764160,"available":95947493376,"internally_reserved":0,"allocated":172040192,"data_stored":171817676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675192,"kb_used_data":170600,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93696648,"statfs":{"total":96636764160,"available":95945367552,"internally_reserved":0,"allocated":174694400,"data_stored":174471884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598404,"kb_used_data":169640,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93773436,"statfs":{"total":96636764160,"available":96023998464,"internally_reserved":0,"allocated":173711360,"data_stored":173488844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":439021266},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673648,"kb_used_data":168608,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93698192,"statfs":{"total":96636764160,"available":95946948608,"internally_reserved":0,"allocated":172654592,"data_stored":172432076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,11,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172109824,"data_stored":172109824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173199360,"data_stored":173199360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174272512,"data_stored":174272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171159552,"data_stored":171159552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:34.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:34.965 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:34.966 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:34.992 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-10T20:27:34.055248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344956776,"num_objects":42113,"num_object_clones":0,"num_object_copies":168452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123105,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152356,"ondisk_log_size":152356,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2620360,"kb_used_data":676856,"kb_used_omap":14,"kb_used_meta":1943409,"kb_avail":374867000,"statfs":{"total":386547056640,"available":383863808000,"internally_reserved":0,"allocated":693100544,"data_stored":692210480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1990051382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,16,18,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22929408,"num_objects":-2799,"num_object_clones":0,"num_object_copies":-11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004363"},"pg_stats":[{"pgid":"1.f","version":"42'9510","reported_seq":9796,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.095002+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:33.095002+0000","last_peered":"2024-09-10T20:27:33.095002+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:33.095002+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9510,"log_dups_size":0,"ondisk_log_size":9510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397504,"num_objects":2612,"num_object_clones":0,"num_object_copies":10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9576","reported_seq":9712,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.771587+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:30.771587+0000","last_peered":"2024-09-10T20:27:30.771587+0000","last_clean":"2024-09-10T20:27:30.771587+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:30.771587+0000","last_undegraded":"2024-09-10T20:27:30.771587+0000","last_fullsized":"2024-09-10T20:27:30.771587+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21921792,"num_objects":2676,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9450","reported_seq":9657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.441403+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:29.441403+0000","last_peered":"2024-09-10T20:27:29.441403+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:29.441403+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22035048,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9449,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9729","reported_seq":13903,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.830573+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:30.830573+0000","last_peered":"2024-09-10T20:27:30.830573+0000","last_clean":"2024-09-10T20:27:30.830573+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:30.830573+0000","last_undegraded":"2024-09-10T20:27:30.830573+0000","last_fullsized":"2024-09-10T20:27:30.830573+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21929984,"num_objects":2677,"num_object_clones":0,"num_object_copies":10708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9570","reported_seq":9894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.129001+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:33.129001+0000","last_peered":"2024-09-10T20:27:33.129001+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:33.129001+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21288096,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9547","reported_seq":9772,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.037208+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:33.037208+0000","last_peered":"2024-09-10T20:27:33.037208+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:33.037208+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20897792,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9495","reported_seq":9702,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.092164+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:32.092164+0000","last_peered":"2024-09-10T20:27:32.092164+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:32.092164+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091488,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9659","reported_seq":9831,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.041691+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:32.041691+0000","last_peered":"2024-09-10T20:27:32.041691+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:32.041691+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9659,"log_dups_size":0,"ondisk_log_size":9659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574816,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9402","reported_seq":13378,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:33.055687+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:33.055687+0000","last_peered":"2024-09-10T20:27:33.055687+0000","last_clean":"2024-09-10T20:27:33.055687+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:33.055687+0000","last_undegraded":"2024-09-10T20:27:33.055687+0000","last_fullsized":"2024-09-10T20:27:33.055687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21289552,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9397","reported_seq":13372,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:29.661043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:29.661043+0000","last_peered":"2024-09-10T20:27:29.661043+0000","last_clean":"2024-09-10T20:27:29.661043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:29.661043+0000","last_undegraded":"2024-09-10T20:27:29.661043+0000","last_fullsized":"2024-09-10T20:27:29.661043+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":2745,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9495","reported_seq":9770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.147643+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:33.147643+0000","last_peered":"2024-09-10T20:27:33.147643+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:33.147643+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21442288,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9492,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":762}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9573","reported_seq":9875,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.953824+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:32.953824+0000","last_peered":"2024-09-10T20:27:32.953824+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:32.953824+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21232208,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":732}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9247","reported_seq":9434,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.486733+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:29.486733+0000","last_peered":"2024-09-10T20:27:29.486733+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:29.486733+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9247,"log_dups_size":0,"ondisk_log_size":9247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21166672,"num_objects":2584,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9246,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9528","reported_seq":10456,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.086415+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:33.086415+0000","last_peered":"2024-09-10T20:27:33.086415+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:33.086415+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9528,"log_dups_size":0,"ondisk_log_size":9528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21017760,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1016}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9701","reported_seq":9910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:29.676694+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:29.676694+0000","last_peered":"2024-09-10T20:27:29.676694+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:29.676694+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22508704,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":840}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9477","reported_seq":13577,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.768788+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:30.768788+0000","last_peered":"2024-09-10T20:27:30.768788+0000","last_clean":"2024-09-10T20:27:30.768788+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:30.768788+0000","last_undegraded":"2024-09-10T20:27:30.768788+0000","last_fullsized":"2024-09-10T20:27:30.768788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21676032,"num_objects":2646,"num_object_clones":0,"num_object_copies":10584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":344956776,"num_objects":42113,"num_object_clones":0,"num_object_copies":168452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123105,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":690741248,"data_stored":690741248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152356,"ondisk_log_size":152356,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673116,"kb_used_data":168008,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93698724,"statfs":{"total":96636764160,"available":95947493376,"internally_reserved":0,"allocated":172040192,"data_stored":171817676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675192,"kb_used_data":170600,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93696648,"statfs":{"total":96636764160,"available":95945367552,"internally_reserved":0,"allocated":174694400,"data_stored":174471884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598404,"kb_used_data":169640,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93773436,"statfs":{"total":96636764160,"available":96023998464,"internally_reserved":0,"allocated":173711360,"data_stored":173488844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":439021266},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673648,"kb_used_data":168608,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93698192,"statfs":{"total":96636764160,"available":95946948608,"internally_reserved":0,"allocated":172654592,"data_stored":172432076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,11,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172109824,"data_stored":172109824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173199360,"data_stored":173199360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174272512,"data_stored":174272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171159552,"data_stored":171159552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:34.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:34.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9510", 'reported_seq': 9796, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:33.095002+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:33.095002+0000', 'last_peered': '2024-09-10T20:27:33.095002+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:33.095002+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9510, 'log_dups_size': 0, 'ondisk_log_size': 9510, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21397504, 'num_objects': 2612, 'num_object_clones': 0, 'num_object_copies': 10448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2612, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9510, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 757}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:34.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9450", 'reported_seq': 9657, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:29.441403+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:29.441403+0000', 'last_peered': '2024-09-10T20:27:29.441403+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:29.441403+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9450, 'log_dups_size': 0, 'ondisk_log_size': 9450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22035048, 'num_objects': 2691, 'num_object_clones': 0, 'num_object_copies': 10764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2691, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9449, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 783}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9570", 'reported_seq': 9894, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:33.129001+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:33.129001+0000', 'last_peered': '2024-09-10T20:27:33.129001+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:33.129001+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9570, 'log_dups_size': 0, 'ondisk_log_size': 9570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21288096, 'num_objects': 2599, 'num_object_clones': 0, 'num_object_copies': 10396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9568, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 744}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9547", 'reported_seq': 9772, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:33.037208+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:33.037208+0000', 'last_peered': '2024-09-10T20:27:33.037208+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:33.037208+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9547, 'log_dups_size': 0, 'ondisk_log_size': 9547, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20897792, 'num_objects': 2551, 'num_object_clones': 0, 'num_object_copies': 10204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9547, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 694}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9495", 'reported_seq': 9702, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:32.092164+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:32.092164+0000', 'last_peered': '2024-09-10T20:27:32.092164+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:32.092164+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9495, 'log_dups_size': 0, 'ondisk_log_size': 9495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21091488, 'num_objects': 2575, 'num_object_clones': 0, 'num_object_copies': 10300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9493, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 682}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9659", 'reported_seq': 9831, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:32.041691+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:32.041691+0000', 'last_peered': '2024-09-10T20:27:32.041691+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:32.041691+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9659, 'log_dups_size': 0, 'ondisk_log_size': 9659, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21574816, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9657, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 634}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:34.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9495", 'reported_seq': 9770, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:33.147643+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:33.147643+0000', 'last_peered': '2024-09-10T20:27:33.147643+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:33.147643+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9495, 'log_dups_size': 0, 'ondisk_log_size': 9495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21442288, 'num_objects': 2618, 'num_object_clones': 0, 'num_object_copies': 10472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9492, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 762}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9573", 'reported_seq': 9875, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:32.953824+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:32.953824+0000', 'last_peered': '2024-09-10T20:27:32.953824+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:32.953824+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9573, 'log_dups_size': 0, 'ondisk_log_size': 9573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21232208, 'num_objects': 2592, 'num_object_clones': 0, 'num_object_copies': 10368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9572, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9247", 'reported_seq': 9434, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:29.486733+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:29.486733+0000', 'last_peered': '2024-09-10T20:27:29.486733+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:29.486733+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9247, 'log_dups_size': 0, 'ondisk_log_size': 9247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21166672, 'num_objects': 2584, 'num_object_clones': 0, 'num_object_copies': 10336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9246, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 681}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9528", 'reported_seq': 10456, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:33.086415+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:33.086415+0000', 'last_peered': '2024-09-10T20:27:33.086415+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:33.086415+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9528, 'log_dups_size': 0, 'ondisk_log_size': 9528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21017760, 'num_objects': 2566, 'num_object_clones': 0, 'num_object_copies': 10264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1550, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9526, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 429, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1016}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:34.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9701", 'reported_seq': 9910, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:29.676694+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:29.676694+0000', 'last_peered': '2024-09-10T20:27:29.676694+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:29.676694+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9701, 'log_dups_size': 0, 'ondisk_log_size': 9701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22508704, 'num_objects': 2748, 'num_object_clones': 0, 'num_object_copies': 10992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9699, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 840}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:34.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:35.761 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:35.761 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20459/168452 objects degraded (12.145%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":544,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42113,"data_bytes":344956776,"bytes_used":2683248640,"bytes_avail":383863808000,"bytes_total":386547056640,"degraded_objects":20459,"degraded_total":168452,"degraded_ratio":0.12145299551207465,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":233},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-10T20:27:32.054445+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:27:35.785 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:35.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:36.390 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:36.390 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:36.410 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-10T20:27:36.056104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":342581096,"num_objects":41823,"num_object_clones":0,"num_object_copies":167292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123395,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152646,"ondisk_log_size":152646,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2638052,"kb_used_data":672340,"kb_used_omap":14,"kb_used_meta":1965617,"kb_avail":374849308,"statfs":{"total":386547056640,"available":383845691392,"internally_reserved":0,"allocated":688476160,"data_stored":687586096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2012792374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,8,5,16,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20283392,"num_objects":-2476,"num_object_clones":0,"num_object_copies":-9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004414"},"pg_stats":[{"pgid":"1.f","version":"42'9510","reported_seq":9796,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.095002+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:33.095002+0000","last_peered":"2024-09-10T20:27:33.095002+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:33.095002+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9510,"log_dups_size":0,"ondisk_log_size":9510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397504,"num_objects":2612,"num_object_clones":0,"num_object_copies":10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9576","reported_seq":9712,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.771587+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:30.771587+0000","last_peered":"2024-09-10T20:27:30.771587+0000","last_clean":"2024-09-10T20:27:30.771587+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:30.771587+0000","last_undegraded":"2024-09-10T20:27:30.771587+0000","last_fullsized":"2024-09-10T20:27:30.771587+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21921792,"num_objects":2676,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9526","reported_seq":9732,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:34.867371+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:34.867371+0000","last_peered":"2024-09-10T20:27:34.867371+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:34.867371+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21412456,"num_objects":2615,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9525,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":707}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9729","reported_seq":13903,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.830573+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:30.830573+0000","last_peered":"2024-09-10T20:27:30.830573+0000","last_clean":"2024-09-10T20:27:30.830573+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:30.830573+0000","last_undegraded":"2024-09-10T20:27:30.830573+0000","last_fullsized":"2024-09-10T20:27:30.830573+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21929984,"num_objects":2677,"num_object_clones":0,"num_object_copies":10708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9570","reported_seq":9894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.129001+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:33.129001+0000","last_peered":"2024-09-10T20:27:33.129001+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:33.129001+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21288096,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9547","reported_seq":9772,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.037208+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:33.037208+0000","last_peered":"2024-09-10T20:27:33.037208+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:33.037208+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20897792,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9495","reported_seq":9702,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.092164+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:32.092164+0000","last_peered":"2024-09-10T20:27:32.092164+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:32.092164+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091488,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9659","reported_seq":9831,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.041691+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:32.041691+0000","last_peered":"2024-09-10T20:27:32.041691+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:32.041691+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9659,"log_dups_size":0,"ondisk_log_size":9659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574816,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9402","reported_seq":13378,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:33.055687+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:33.055687+0000","last_peered":"2024-09-10T20:27:33.055687+0000","last_clean":"2024-09-10T20:27:33.055687+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:33.055687+0000","last_undegraded":"2024-09-10T20:27:33.055687+0000","last_fullsized":"2024-09-10T20:27:33.055687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21289552,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9463","reported_seq":13438,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:34.728923+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:34.728923+0000","last_peered":"2024-09-10T20:27:34.728923+0000","last_clean":"2024-09-10T20:27:34.728923+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:34.728923+0000","last_undegraded":"2024-09-10T20:27:34.728923+0000","last_fullsized":"2024-09-10T20:27:34.728923+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21946368,"num_objects":2679,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9495","reported_seq":9770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.147643+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:33.147643+0000","last_peered":"2024-09-10T20:27:33.147643+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:33.147643+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21442288,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9492,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":762}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9573","reported_seq":9875,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.953824+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:32.953824+0000","last_peered":"2024-09-10T20:27:32.953824+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:32.953824+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21232208,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":732}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9329","reported_seq":9516,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:34.725978+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:34.725978+0000","last_peered":"2024-09-10T20:27:34.725978+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:34.725978+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20494928,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":599}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9528","reported_seq":10456,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.086415+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:33.086415+0000","last_peered":"2024-09-10T20:27:33.086415+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:33.086415+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9528,"log_dups_size":0,"ondisk_log_size":9528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21017760,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1016}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9767","reported_seq":9976,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:34.833780+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:34.833780+0000","last_peered":"2024-09-10T20:27:34.833780+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:34.833780+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9767,"log_dups_size":0,"ondisk_log_size":9767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21968032,"num_objects":2682,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9765,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":774}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9477","reported_seq":13577,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:30.768788+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:30.768788+0000","last_peered":"2024-09-10T20:27:30.768788+0000","last_clean":"2024-09-10T20:27:30.768788+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:30.768788+0000","last_undegraded":"2024-09-10T20:27:30.768788+0000","last_fullsized":"2024-09-10T20:27:30.768788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21676032,"num_objects":2646,"num_object_clones":0,"num_object_copies":10584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":342581096,"num_objects":41823,"num_object_clones":0,"num_object_copies":167292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123395,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":686153728,"data_stored":686153728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152646,"ondisk_log_size":152646,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673116,"kb_used_data":168008,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93698724,"statfs":{"total":96636764160,"available":95947493376,"internally_reserved":0,"allocated":172040192,"data_stored":171817676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692884,"kb_used_data":166084,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93678956,"statfs":{"total":96636764160,"available":95927250944,"internally_reserved":0,"allocated":170070016,"data_stored":169847500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598404,"kb_used_data":169640,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93773436,"statfs":{"total":96636764160,"available":96023998464,"internally_reserved":0,"allocated":173711360,"data_stored":173488844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":439021266},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673648,"kb_used_data":168608,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93698192,"statfs":{"total":96636764160,"available":95946948608,"internally_reserved":0,"allocated":172654592,"data_stored":172432076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,11,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172109824,"data_stored":172109824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173199360,"data_stored":173199360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169684992,"data_stored":169684992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171159552,"data_stored":171159552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:39.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:40.033 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:40.067 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-10T20:27:38.056744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339836776,"num_objects":41488,"num_object_clones":0,"num_object_copies":165952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123730,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152981,"ondisk_log_size":152981,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2655788,"kb_used_data":663720,"kb_used_omap":14,"kb_used_meta":1991985,"kb_avail":374831572,"statfs":{"total":386547056640,"available":383827529728,"internally_reserved":0,"allocated":679649280,"data_stored":678759216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2039793206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,6,6,7,8,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19226624,"num_objects":-2347,"num_object_clones":0,"num_object_copies":-9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004424"},"pg_stats":[{"pgid":"1.f","version":"42'9510","reported_seq":9796,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.095002+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:33.095002+0000","last_peered":"2024-09-10T20:27:33.095002+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:33.095002+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9510,"log_dups_size":0,"ondisk_log_size":9510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397504,"num_objects":2612,"num_object_clones":0,"num_object_copies":10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9642","reported_seq":9777,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:35.756457+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:35.756457+0000","last_peered":"2024-09-10T20:27:35.756457+0000","last_clean":"2024-09-10T20:27:35.756457+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:35.756457+0000","last_undegraded":"2024-09-10T20:27:35.756457+0000","last_fullsized":"2024-09-10T20:27:35.756457+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21381120,"num_objects":2610,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9526","reported_seq":9732,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:34.867371+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:34.867371+0000","last_peered":"2024-09-10T20:27:34.867371+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:34.867371+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21412456,"num_objects":2615,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9525,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":707}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9807","reported_seq":13981,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:35.970828+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:35.970828+0000","last_peered":"2024-09-10T20:27:35.970828+0000","last_clean":"2024-09-10T20:27:35.970828+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:35.970828+0000","last_undegraded":"2024-09-10T20:27:35.970828+0000","last_fullsized":"2024-09-10T20:27:35.970828+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9807,"log_dups_size":0,"ondisk_log_size":9807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21291008,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4067,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9570","reported_seq":9894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.129001+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:33.129001+0000","last_peered":"2024-09-10T20:27:33.129001+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:33.129001+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21288096,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9547","reported_seq":9772,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.037208+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:33.037208+0000","last_peered":"2024-09-10T20:27:33.037208+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:33.037208+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20897792,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9560","reported_seq":9767,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:37.053247+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:37.053247+0000","last_peered":"2024-09-10T20:27:37.053247+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:37.053247+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20559008,"num_objects":2510,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9558,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9715","reported_seq":9886,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:37.154357+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:37.154357+0000","last_peered":"2024-09-10T20:27:37.154357+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:37.154357+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21116064,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9402","reported_seq":13378,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:33.055687+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:33.055687+0000","last_peered":"2024-09-10T20:27:33.055687+0000","last_clean":"2024-09-10T20:27:33.055687+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:33.055687+0000","last_undegraded":"2024-09-10T20:27:33.055687+0000","last_fullsized":"2024-09-10T20:27:33.055687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21289552,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9463","reported_seq":13438,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:34.728923+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:34.728923+0000","last_peered":"2024-09-10T20:27:34.728923+0000","last_clean":"2024-09-10T20:27:34.728923+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:34.728923+0000","last_undegraded":"2024-09-10T20:27:34.728923+0000","last_fullsized":"2024-09-10T20:27:34.728923+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21946368,"num_objects":2679,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9495","reported_seq":9770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.147643+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:33.147643+0000","last_peered":"2024-09-10T20:27:33.147643+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:33.147643+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21442288,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9492,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":762}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9573","reported_seq":9875,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:32.953824+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:32.953824+0000","last_peered":"2024-09-10T20:27:32.953824+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:32.953824+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21232208,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":732}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9329","reported_seq":9516,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:34.725978+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:34.725978+0000","last_peered":"2024-09-10T20:27:34.725978+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:34.725978+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20494928,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":599}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9528","reported_seq":10456,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:33.086415+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:33.086415+0000","last_peered":"2024-09-10T20:27:33.086415+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:33.086415+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9528,"log_dups_size":0,"ondisk_log_size":9528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21017760,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1016}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9767","reported_seq":9976,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:34.833780+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:34.833780+0000","last_peered":"2024-09-10T20:27:34.833780+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:34.833780+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9767,"log_dups_size":0,"ondisk_log_size":9767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21968032,"num_objects":2682,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9765,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":774}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9547","reported_seq":13647,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:36.009532+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:36.009532+0000","last_peered":"2024-09-10T20:27:36.009532+0000","last_clean":"2024-09-10T20:27:36.009532+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:36.009532+0000","last_undegraded":"2024-09-10T20:27:36.009532+0000","last_fullsized":"2024-09-10T20:27:36.009532+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102592,"num_objects":2576,"num_object_clones":0,"num_object_copies":10304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339836776,"num_objects":41488,"num_object_clones":0,"num_object_copies":165952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123730,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":677261312,"data_stored":677261312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152981,"ondisk_log_size":152981,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673116,"kb_used_data":168008,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93698724,"statfs":{"total":96636764160,"available":95947493376,"internally_reserved":0,"allocated":172040192,"data_stored":171817676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692884,"kb_used_data":166084,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93678956,"statfs":{"total":96636764160,"available":95927250944,"internally_reserved":0,"allocated":170070016,"data_stored":169847500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598180,"kb_used_data":165268,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93773660,"statfs":{"total":96636764160,"available":96024227840,"internally_reserved":0,"allocated":169234432,"data_stored":169011916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,2,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691608,"kb_used_data":164360,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93680232,"statfs":{"total":96636764160,"available":95928557568,"internally_reserved":0,"allocated":168304640,"data_stored":168082124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539881167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167682048,"data_stored":167682048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168734720,"data_stored":168734720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169684992,"data_stored":169684992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171159552,"data_stored":171159552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:40.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:40.674 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:40.675 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:40.698 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-10T20:27:40.057279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":333643624,"num_objects":40732,"num_object_clones":0,"num_object_copies":162928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153737,"ondisk_log_size":153737,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2652700,"kb_used_data":655056,"kb_used_omap":14,"kb_used_meta":1997553,"kb_avail":374834660,"statfs":{"total":386547056640,"available":383830691840,"internally_reserved":0,"allocated":670777344,"data_stored":669887280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2045494838},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,0,2,11,15,10,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24199168,"num_objects":-2954,"num_object_clones":0,"num_object_copies":-11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004455"},"pg_stats":[{"pgid":"1.f","version":"42'9574","reported_seq":9860,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.234749+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:38.234749+0000","last_peered":"2024-09-10T20:27:38.234749+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:38.234749+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20873216,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9642","reported_seq":9777,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:35.756457+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:35.756457+0000","last_peered":"2024-09-10T20:27:35.756457+0000","last_clean":"2024-09-10T20:27:35.756457+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:35.756457+0000","last_undegraded":"2024-09-10T20:27:35.756457+0000","last_fullsized":"2024-09-10T20:27:35.756457+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21381120,"num_objects":2610,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9594","reported_seq":9801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.741694+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:39.741694+0000","last_peered":"2024-09-10T20:27:39.741694+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:39.741694+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9594,"log_dups_size":0,"ondisk_log_size":9594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20855400,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9593,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9807","reported_seq":13981,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:35.970828+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:35.970828+0000","last_peered":"2024-09-10T20:27:35.970828+0000","last_clean":"2024-09-10T20:27:35.970828+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:35.970828+0000","last_undegraded":"2024-09-10T20:27:35.970828+0000","last_fullsized":"2024-09-10T20:27:35.970828+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9807,"log_dups_size":0,"ondisk_log_size":9807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21291008,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4067,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9647","reported_seq":9971,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.276047+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:38.276047+0000","last_peered":"2024-09-10T20:27:38.276047+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:38.276047+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9647,"log_dups_size":0,"ondisk_log_size":9647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9645,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9607","reported_seq":9833,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.169806+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:38.169806+0000","last_peered":"2024-09-10T20:27:38.169806+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:38.169806+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20406272,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9560","reported_seq":9767,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:37.053247+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:37.053247+0000","last_peered":"2024-09-10T20:27:37.053247+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:37.053247+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20559008,"num_objects":2510,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9558,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9715","reported_seq":9886,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:37.154357+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:37.154357+0000","last_peered":"2024-09-10T20:27:37.154357+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:37.154357+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21116064,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9469","reported_seq":13445,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:38.309666+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:38.309666+0000","last_peered":"2024-09-10T20:27:38.309666+0000","last_clean":"2024-09-10T20:27:38.309666+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:38.309666+0000","last_undegraded":"2024-09-10T20:27:38.309666+0000","last_fullsized":"2024-09-10T20:27:38.309666+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20740688,"num_objects":2532,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9525","reported_seq":13500,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:39.577217+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:39.577217+0000","last_peered":"2024-09-10T20:27:39.577217+0000","last_clean":"2024-09-10T20:27:39.577217+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:39.577217+0000","last_undegraded":"2024-09-10T20:27:39.577217+0000","last_fullsized":"2024-09-10T20:27:39.577217+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21438464,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9566","reported_seq":9843,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.181500+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:38.181500+0000","last_peered":"2024-09-10T20:27:38.181500+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:38.181500+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9566,"log_dups_size":0,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20860656,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9636","reported_seq":9938,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.258640+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:38.258640+0000","last_peered":"2024-09-10T20:27:38.258640+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:38.258640+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20716112,"num_objects":2529,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9635,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9388","reported_seq":9575,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.781066+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:39.781066+0000","last_peered":"2024-09-10T20:27:39.781066+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:39.781066+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20011600,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9609","reported_seq":10617,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.176527+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:38.176527+0000","last_peered":"2024-09-10T20:27:38.176527+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:38.176527+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20354208,"num_objects":2485,"num_object_clones":0,"num_object_copies":9940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9851","reported_seq":10059,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.895431+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:39.895431+0000","last_peered":"2024-09-10T20:27:39.895431+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:39.895431+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21279904,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9849,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9547","reported_seq":13647,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:36.009532+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:36.009532+0000","last_peered":"2024-09-10T20:27:36.009532+0000","last_clean":"2024-09-10T20:27:36.009532+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:36.009532+0000","last_undegraded":"2024-09-10T20:27:36.009532+0000","last_fullsized":"2024-09-10T20:27:36.009532+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102592,"num_objects":2576,"num_object_clones":0,"num_object_copies":10304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":333643624,"num_objects":40732,"num_object_clones":0,"num_object_copies":162928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":668160000,"data_stored":668160000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153737,"ondisk_log_size":153737,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117095,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686876,"kb_used_data":163712,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93684964,"statfs":{"total":96636764160,"available":95933403136,"internally_reserved":0,"allocated":167641088,"data_stored":167418572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":535686864},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,6,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676036,"kb_used_data":161716,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93695804,"statfs":{"total":96636764160,"available":95944503296,"internally_reserved":0,"allocated":165597184,"data_stored":165374668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598180,"kb_used_data":165268,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93773660,"statfs":{"total":96636764160,"available":96024227840,"internally_reserved":0,"allocated":169234432,"data_stored":169011916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,2,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691608,"kb_used_data":164360,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93680232,"statfs":{"total":96636764160,"available":95928557568,"internally_reserved":0,"allocated":168304640,"data_stored":168082124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539881167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167682048,"data_stored":167682048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168734720,"data_stored":168734720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165175296,"data_stored":165175296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":166567936,"data_stored":166567936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:40.699 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:40.699 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9574", 'reported_seq': 9860, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:38.234749+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:38.234749+0000', 'last_peered': '2024-09-10T20:27:38.234749+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:38.234749+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9574, 'log_dups_size': 0, 'ondisk_log_size': 9574, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20873216, 'num_objects': 2548, 'num_object_clones': 0, 'num_object_copies': 10192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9574, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 693}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:40.699 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:40.699 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9594", 'reported_seq': 9801, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:39.741694+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:39.741694+0000', 'last_peered': '2024-09-10T20:27:39.741694+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:39.741694+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9594, 'log_dups_size': 0, 'ondisk_log_size': 9594, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20855400, 'num_objects': 2547, 'num_object_clones': 0, 'num_object_copies': 10188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2547, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9593, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 639}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:40.699 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:40.699 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9647", 'reported_seq': 9971, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:38.276047+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:38.276047+0000', 'last_peered': '2024-09-10T20:27:38.276047+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:38.276047+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9647, 'log_dups_size': 0, 'ondisk_log_size': 9647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20657312, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9645, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 667}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9607", 'reported_seq': 9833, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:38.169806+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:38.169806+0000', 'last_peered': '2024-09-10T20:27:38.169806+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:38.169806+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9607, 'log_dups_size': 0, 'ondisk_log_size': 9607, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20406272, 'num_objects': 2491, 'num_object_clones': 0, 'num_object_copies': 9964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9607, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 634}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9560", 'reported_seq': 9767, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:37.053247+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:37.053247+0000', 'last_peered': '2024-09-10T20:27:37.053247+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:37.053247+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9560, 'log_dups_size': 0, 'ondisk_log_size': 9560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20559008, 'num_objects': 2510, 'num_object_clones': 0, 'num_object_copies': 10040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9558, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 617}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9715", 'reported_seq': 9886, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:37.154357+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:37.154357+0000', 'last_peered': '2024-09-10T20:27:37.154357+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:37.154357+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9715, 'log_dups_size': 0, 'ondisk_log_size': 9715, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21116064, 'num_objects': 2578, 'num_object_clones': 0, 'num_object_copies': 10312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9713, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9566", 'reported_seq': 9843, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:38.181500+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:38.181500+0000', 'last_peered': '2024-09-10T20:27:38.181500+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:38.181500+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9566, 'log_dups_size': 0, 'ondisk_log_size': 9566, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20860656, 'num_objects': 2547, 'num_object_clones': 0, 'num_object_copies': 10188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9563, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 691}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9636", 'reported_seq': 9938, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:38.258640+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:38.258640+0000', 'last_peered': '2024-09-10T20:27:38.258640+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:38.258640+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9636, 'log_dups_size': 0, 'ondisk_log_size': 9636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20716112, 'num_objects': 2529, 'num_object_clones': 0, 'num_object_copies': 10116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9635, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 669}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:40.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:40.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9388", 'reported_seq': 9575, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:39.781066+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:39.781066+0000', 'last_peered': '2024-09-10T20:27:39.781066+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:39.781066+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9388, 'log_dups_size': 0, 'ondisk_log_size': 9388, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20011600, 'num_objects': 2443, 'num_object_clones': 0, 'num_object_copies': 9772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9387, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 540}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:40.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:40.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9609", 'reported_seq': 10617, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:38.176527+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:38.176527+0000', 'last_peered': '2024-09-10T20:27:38.176527+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:38.176527+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9609, 'log_dups_size': 0, 'ondisk_log_size': 9609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20354208, 'num_objects': 2485, 'num_object_clones': 0, 'num_object_copies': 9940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1510, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9607, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 469, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:40.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:40.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9851", 'reported_seq': 10059, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:39.895431+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:39.895431+0000', 'last_peered': '2024-09-10T20:27:39.895431+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:39.895431+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9851, 'log_dups_size': 0, 'ondisk_log_size': 9851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21279904, 'num_objects': 2598, 'num_object_clones': 0, 'num_object_copies': 10392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9849, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:40.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:41.343 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:41.343 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20419/162928 objects degraded (12.533%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":550,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40732,"data_bytes":333643624,"bytes_used":2716364800,"bytes_avail":383830691840,"bytes_total":386547056640,"degraded_objects":20419,"degraded_total":162928,"degraded_ratio":0.12532529706373369,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":246},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-10T20:27:40.057375+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:27:41.366 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:41.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:41.978 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:41.979 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:42.006 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-10T20:27:40.057279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":333643624,"num_objects":40732,"num_object_clones":0,"num_object_copies":162928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153737,"ondisk_log_size":153737,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2652700,"kb_used_data":655056,"kb_used_omap":14,"kb_used_meta":1997553,"kb_avail":374834660,"statfs":{"total":386547056640,"available":383830691840,"internally_reserved":0,"allocated":670777344,"data_stored":669887280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2045494838},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,0,2,11,15,10,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24199168,"num_objects":-2954,"num_object_clones":0,"num_object_copies":-11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004455"},"pg_stats":[{"pgid":"1.f","version":"42'9574","reported_seq":9860,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.234749+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:38.234749+0000","last_peered":"2024-09-10T20:27:38.234749+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:38.234749+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20873216,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9642","reported_seq":9777,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:35.756457+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:35.756457+0000","last_peered":"2024-09-10T20:27:35.756457+0000","last_clean":"2024-09-10T20:27:35.756457+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:35.756457+0000","last_undegraded":"2024-09-10T20:27:35.756457+0000","last_fullsized":"2024-09-10T20:27:35.756457+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21381120,"num_objects":2610,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9594","reported_seq":9801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.741694+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:39.741694+0000","last_peered":"2024-09-10T20:27:39.741694+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:39.741694+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9594,"log_dups_size":0,"ondisk_log_size":9594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20855400,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9593,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9807","reported_seq":13981,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:35.970828+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:35.970828+0000","last_peered":"2024-09-10T20:27:35.970828+0000","last_clean":"2024-09-10T20:27:35.970828+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:35.970828+0000","last_undegraded":"2024-09-10T20:27:35.970828+0000","last_fullsized":"2024-09-10T20:27:35.970828+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9807,"log_dups_size":0,"ondisk_log_size":9807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21291008,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4067,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9647","reported_seq":9971,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.276047+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:38.276047+0000","last_peered":"2024-09-10T20:27:38.276047+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:38.276047+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9647,"log_dups_size":0,"ondisk_log_size":9647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9645,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9607","reported_seq":9833,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.169806+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:38.169806+0000","last_peered":"2024-09-10T20:27:38.169806+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:38.169806+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20406272,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9560","reported_seq":9767,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:37.053247+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:37.053247+0000","last_peered":"2024-09-10T20:27:37.053247+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:37.053247+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20559008,"num_objects":2510,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9558,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9715","reported_seq":9886,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:37.154357+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:37.154357+0000","last_peered":"2024-09-10T20:27:37.154357+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:37.154357+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21116064,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9469","reported_seq":13445,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:38.309666+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:38.309666+0000","last_peered":"2024-09-10T20:27:38.309666+0000","last_clean":"2024-09-10T20:27:38.309666+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:38.309666+0000","last_undegraded":"2024-09-10T20:27:38.309666+0000","last_fullsized":"2024-09-10T20:27:38.309666+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20740688,"num_objects":2532,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9525","reported_seq":13500,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:39.577217+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:39.577217+0000","last_peered":"2024-09-10T20:27:39.577217+0000","last_clean":"2024-09-10T20:27:39.577217+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:39.577217+0000","last_undegraded":"2024-09-10T20:27:39.577217+0000","last_fullsized":"2024-09-10T20:27:39.577217+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21438464,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9566","reported_seq":9843,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.181500+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:38.181500+0000","last_peered":"2024-09-10T20:27:38.181500+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:38.181500+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9566,"log_dups_size":0,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20860656,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9636","reported_seq":9938,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.258640+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:38.258640+0000","last_peered":"2024-09-10T20:27:38.258640+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:38.258640+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20716112,"num_objects":2529,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9635,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9388","reported_seq":9575,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.781066+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:39.781066+0000","last_peered":"2024-09-10T20:27:39.781066+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:39.781066+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20011600,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9609","reported_seq":10617,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:38.176527+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:38.176527+0000","last_peered":"2024-09-10T20:27:38.176527+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:38.176527+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20354208,"num_objects":2485,"num_object_clones":0,"num_object_copies":9940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9851","reported_seq":10059,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.895431+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:39.895431+0000","last_peered":"2024-09-10T20:27:39.895431+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:39.895431+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21279904,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9849,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9547","reported_seq":13647,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:36.009532+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:36.009532+0000","last_peered":"2024-09-10T20:27:36.009532+0000","last_clean":"2024-09-10T20:27:36.009532+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:36.009532+0000","last_undegraded":"2024-09-10T20:27:36.009532+0000","last_fullsized":"2024-09-10T20:27:36.009532+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102592,"num_objects":2576,"num_object_clones":0,"num_object_copies":10304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":333643624,"num_objects":40732,"num_object_clones":0,"num_object_copies":162928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":668160000,"data_stored":668160000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153737,"ondisk_log_size":153737,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117095,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686876,"kb_used_data":163712,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93684964,"statfs":{"total":96636764160,"available":95933403136,"internally_reserved":0,"allocated":167641088,"data_stored":167418572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":535686864},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,6,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676036,"kb_used_data":161716,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93695804,"statfs":{"total":96636764160,"available":95944503296,"internally_reserved":0,"allocated":165597184,"data_stored":165374668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598180,"kb_used_data":165268,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93773660,"statfs":{"total":96636764160,"available":96024227840,"internally_reserved":0,"allocated":169234432,"data_stored":169011916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,2,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691608,"kb_used_data":164360,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93680232,"statfs":{"total":96636764160,"available":95928557568,"internally_reserved":0,"allocated":168304640,"data_stored":168082124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539881167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167682048,"data_stored":167682048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168734720,"data_stored":168734720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165175296,"data_stored":165175296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":166567936,"data_stored":166567936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:45.008 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:45.619 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:45.620 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:45.639 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-10T20:27:44.058534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":327098216,"num_objects":39933,"num_object_clones":0,"num_object_copies":159732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125285,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154536,"ondisk_log_size":154536,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670560,"kb_used_data":642400,"kb_used_omap":14,"kb_used_meta":2028081,"kb_avail":374816800,"statfs":{"total":386547056640,"available":383812403200,"internally_reserved":0,"allocated":657817600,"data_stored":656927536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2076755510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,10,1,9,11,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22986752,"num_objects":-2806,"num_object_clones":0,"num_object_copies":-11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004179"},"pg_stats":[{"pgid":"1.f","version":"42'9650","reported_seq":9933,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.313689+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:43.313689+0000","last_peered":"2024-09-10T20:27:43.313689+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:43.313689+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9712","reported_seq":9848,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.074400+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:41.074400+0000","last_peered":"2024-09-10T20:27:41.074400+0000","last_clean":"2024-09-10T20:27:41.074400+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:41.074400+0000","last_undegraded":"2024-09-10T20:27:41.074400+0000","last_fullsized":"2024-09-10T20:27:41.074400+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9712,"log_dups_size":0,"ondisk_log_size":9712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20807680,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9594","reported_seq":9801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.741694+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:39.741694+0000","last_peered":"2024-09-10T20:27:39.741694+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:39.741694+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9594,"log_dups_size":0,"ondisk_log_size":9594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20855400,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9593,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9878","reported_seq":14052,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.064951+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:41.064951+0000","last_peered":"2024-09-10T20:27:41.064951+0000","last_clean":"2024-09-10T20:27:41.064951+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:41.064951+0000","last_undegraded":"2024-09-10T20:27:41.064951+0000","last_fullsized":"2024-09-10T20:27:41.064951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9718","reported_seq":10041,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.365575+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:43.365575+0000","last_peered":"2024-09-10T20:27:43.365575+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:43.365575+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9718,"log_dups_size":0,"ondisk_log_size":9718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075680,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9668","reported_seq":9894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.278012+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:43.278012+0000","last_peered":"2024-09-10T20:27:43.278012+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:43.278012+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19906560,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9631","reported_seq":9839,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:42.173275+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:42.173275+0000","last_peered":"2024-09-10T20:27:42.173275+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:42.173275+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19977376,"num_objects":2439,"num_object_clones":0,"num_object_copies":9756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":546}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9787","reported_seq":9958,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:42.133919+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:42.133919+0000","last_peered":"2024-09-10T20:27:42.133919+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:42.133919+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9787,"log_dups_size":0,"ondisk_log_size":9787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20526240,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9532","reported_seq":13508,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:43.365510+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:43.365510+0000","last_peered":"2024-09-10T20:27:43.365510+0000","last_clean":"2024-09-10T20:27:43.365510+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:43.365510+0000","last_undegraded":"2024-09-10T20:27:43.365510+0000","last_fullsized":"2024-09-10T20:27:43.365510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9532,"log_dups_size":0,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20224592,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9525","reported_seq":13500,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:39.577217+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:39.577217+0000","last_peered":"2024-09-10T20:27:39.577217+0000","last_clean":"2024-09-10T20:27:39.577217+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:39.577217+0000","last_undegraded":"2024-09-10T20:27:39.577217+0000","last_fullsized":"2024-09-10T20:27:39.577217+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21438464,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9619","reported_seq":9896,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.183702+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:43.183702+0000","last_peered":"2024-09-10T20:27:43.183702+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:43.183702+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9619,"log_dups_size":0,"ondisk_log_size":9619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20426480,"num_objects":2494,"num_object_clones":0,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9703","reported_seq":10005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.289446+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:43.289446+0000","last_peered":"2024-09-10T20:27:43.289446+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:43.289446+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20167248,"num_objects":2462,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":602}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9388","reported_seq":9575,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.781066+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:39.781066+0000","last_peered":"2024-09-10T20:27:39.781066+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:39.781066+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20011600,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9675","reported_seq":10769,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.324422+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:43.324422+0000","last_peered":"2024-09-10T20:27:43.324422+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:43.324422+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19813536,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9673,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9851","reported_seq":10059,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:39.895431+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:39.895431+0000","last_peered":"2024-09-10T20:27:39.895431+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:39.895431+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21279904,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9849,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9605","reported_seq":13705,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.133771+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:41.133771+0000","last_peered":"2024-09-10T20:27:41.133771+0000","last_clean":"2024-09-10T20:27:41.133771+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:41.133771+0000","last_undegraded":"2024-09-10T20:27:41.133771+0000","last_fullsized":"2024-09-10T20:27:41.133771+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":478,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":327098216,"num_objects":39933,"num_object_clones":0,"num_object_copies":159732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125285,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655147008,"data_stored":655147008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154536,"ondisk_log_size":154536,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117096,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704940,"kb_used_data":159580,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93666900,"statfs":{"total":96636764160,"available":95914905600,"internally_reserved":0,"allocated":163409920,"data_stored":163187404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676036,"kb_used_data":161716,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93695804,"statfs":{"total":96636764160,"available":95944503296,"internally_reserved":0,"allocated":165597184,"data_stored":165374668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598024,"kb_used_data":160952,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93773816,"statfs":{"total":96636764160,"available":96024387584,"internally_reserved":0,"allocated":164814848,"data_stored":164592332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447540946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691560,"kb_used_data":160152,"kb_used_omap":4,"kb_used_meta":531387,"kb_avail":93680280,"statfs":{"total":96636764160,"available":95928606720,"internally_reserved":0,"allocated":163995648,"data_stored":163773132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":544141007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163397632,"data_stored":163397632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164306944,"data_stored":164306944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165175296,"data_stored":165175296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":162267136,"data_stored":162267136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:45.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:46.273 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:46.294 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-10T20:27:46.059217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":324788072,"num_objects":39651,"num_object_clones":0,"num_object_copies":158604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125567,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154818,"ondisk_log_size":154818,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2689372,"kb_used_data":639004,"kb_used_omap":14,"kb_used_meta":2050289,"kb_avail":374797988,"statfs":{"total":386547056640,"available":383793139712,"internally_reserved":0,"allocated":654340096,"data_stored":653450032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2099496502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,10,0,5,9,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20168704,"num_objects":-2462,"num_object_clones":0,"num_object_copies":-9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2462,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003968"},"pg_stats":[{"pgid":"1.f","version":"42'9650","reported_seq":9933,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.313689+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:43.313689+0000","last_peered":"2024-09-10T20:27:43.313689+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:43.313689+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9712","reported_seq":9848,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.074400+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:41.074400+0000","last_peered":"2024-09-10T20:27:41.074400+0000","last_clean":"2024-09-10T20:27:41.074400+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:41.074400+0000","last_undegraded":"2024-09-10T20:27:41.074400+0000","last_fullsized":"2024-09-10T20:27:41.074400+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9712,"log_dups_size":0,"ondisk_log_size":9712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20807680,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9667","reported_seq":9872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:45.022504+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:45.022504+0000","last_peered":"2024-09-10T20:27:45.022504+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:45.022504+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20257384,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9666,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9878","reported_seq":14052,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.064951+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:41.064951+0000","last_peered":"2024-09-10T20:27:41.064951+0000","last_clean":"2024-09-10T20:27:41.064951+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:41.064951+0000","last_undegraded":"2024-09-10T20:27:41.064951+0000","last_fullsized":"2024-09-10T20:27:41.064951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9718","reported_seq":10041,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.365575+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:43.365575+0000","last_peered":"2024-09-10T20:27:43.365575+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:43.365575+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9718,"log_dups_size":0,"ondisk_log_size":9718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075680,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9668","reported_seq":9894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.278012+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:43.278012+0000","last_peered":"2024-09-10T20:27:43.278012+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:43.278012+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19906560,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9631","reported_seq":9839,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:42.173275+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:42.173275+0000","last_peered":"2024-09-10T20:27:42.173275+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:42.173275+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19977376,"num_objects":2439,"num_object_clones":0,"num_object_copies":9756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":546}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9787","reported_seq":9958,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:42.133919+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:42.133919+0000","last_peered":"2024-09-10T20:27:42.133919+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:42.133919+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9787,"log_dups_size":0,"ondisk_log_size":9787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20526240,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9532","reported_seq":13508,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:43.365510+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:43.365510+0000","last_peered":"2024-09-10T20:27:43.365510+0000","last_clean":"2024-09-10T20:27:43.365510+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:43.365510+0000","last_undegraded":"2024-09-10T20:27:43.365510+0000","last_fullsized":"2024-09-10T20:27:43.365510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9532,"log_dups_size":0,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20224592,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9609","reported_seq":13584,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:44.986094+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:44.986094+0000","last_peered":"2024-09-10T20:27:44.986094+0000","last_clean":"2024-09-10T20:27:44.986094+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:44.986094+0000","last_undegraded":"2024-09-10T20:27:44.986094+0000","last_fullsized":"2024-09-10T20:27:44.986094+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20750336,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9619","reported_seq":9896,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.183702+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:43.183702+0000","last_peered":"2024-09-10T20:27:43.183702+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:43.183702+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9619,"log_dups_size":0,"ondisk_log_size":9619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20426480,"num_objects":2494,"num_object_clones":0,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9703","reported_seq":10005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.289446+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:43.289446+0000","last_peered":"2024-09-10T20:27:43.289446+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:43.289446+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20167248,"num_objects":2462,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":602}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9437","reported_seq":9624,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.770816+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:44.770816+0000","last_peered":"2024-09-10T20:27:44.770816+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:44.770816+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9437,"log_dups_size":0,"ondisk_log_size":9437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19610192,"num_objects":2394,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9675","reported_seq":10769,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.324422+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:43.324422+0000","last_peered":"2024-09-10T20:27:43.324422+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:43.324422+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19813536,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9673,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9927","reported_seq":10136,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.876859+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:44.876859+0000","last_peered":"2024-09-10T20:27:44.876859+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:44.876859+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9605","reported_seq":13705,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.133771+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:41.133771+0000","last_peered":"2024-09-10T20:27:41.133771+0000","last_clean":"2024-09-10T20:27:41.133771+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:41.133771+0000","last_undegraded":"2024-09-10T20:27:41.133771+0000","last_fullsized":"2024-09-10T20:27:41.133771+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":478,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":324788072,"num_objects":39651,"num_object_clones":0,"num_object_copies":158604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125567,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":650784768,"data_stored":650784768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154818,"ondisk_log_size":154818,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117096,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704940,"kb_used_data":159580,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93666900,"statfs":{"total":96636764160,"available":95914905600,"internally_reserved":0,"allocated":163409920,"data_stored":163187404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694848,"kb_used_data":158320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93676992,"statfs":{"total":96636764160,"available":95925239808,"internally_reserved":0,"allocated":162119680,"data_stored":161897164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598024,"kb_used_data":160952,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93773816,"statfs":{"total":96636764160,"available":96024387584,"internally_reserved":0,"allocated":164814848,"data_stored":164592332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447540946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691560,"kb_used_data":160152,"kb_used_omap":4,"kb_used_meta":531387,"kb_avail":93680280,"statfs":{"total":96636764160,"available":95928606720,"internally_reserved":0,"allocated":163995648,"data_stored":163773132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":544141007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163397632,"data_stored":163397632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164306944,"data_stored":164306944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160813056,"data_stored":160813056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":162267136,"data_stored":162267136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9650", 'reported_seq': 9933, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:43.313689+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:43.313689+0000', 'last_peered': '2024-09-10T20:27:43.313689+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:43.313689+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9650, 'log_dups_size': 0, 'ondisk_log_size': 9650, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20250624, 'num_objects': 2472, 'num_object_clones': 0, 'num_object_copies': 9888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9650, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 617}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9667", 'reported_seq': 9872, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:45.022504+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:45.022504+0000', 'last_peered': '2024-09-10T20:27:45.022504+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:45.022504+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9667, 'log_dups_size': 0, 'ondisk_log_size': 9667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20257384, 'num_objects': 2474, 'num_object_clones': 0, 'num_object_copies': 9896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2474, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9666, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 566}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9718", 'reported_seq': 10041, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:43.365575+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:43.365575+0000', 'last_peered': '2024-09-10T20:27:43.365575+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:43.365575+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9718, 'log_dups_size': 0, 'ondisk_log_size': 9718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20075680, 'num_objects': 2451, 'num_object_clones': 0, 'num_object_copies': 9804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9716, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 596}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9668", 'reported_seq': 9894, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:43.278012+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:43.278012+0000', 'last_peered': '2024-09-10T20:27:43.278012+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:43.278012+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9668, 'log_dups_size': 0, 'ondisk_log_size': 9668, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19906560, 'num_objects': 2430, 'num_object_clones': 0, 'num_object_copies': 9720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9668, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 573}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9631", 'reported_seq': 9839, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:42.173275+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:42.173275+0000', 'last_peered': '2024-09-10T20:27:42.173275+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:42.173275+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9631, 'log_dups_size': 0, 'ondisk_log_size': 9631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19977376, 'num_objects': 2439, 'num_object_clones': 0, 'num_object_copies': 9756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9629, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 546}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9787", 'reported_seq': 9958, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:42.133919+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:42.133919+0000', 'last_peered': '2024-09-10T20:27:42.133919+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:42.133919+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9787, 'log_dups_size': 0, 'ondisk_log_size': 9787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20526240, 'num_objects': 2506, 'num_object_clones': 0, 'num_object_copies': 10024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9785, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 506}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9619", 'reported_seq': 9896, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:43.183702+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:43.183702+0000', 'last_peered': '2024-09-10T20:27:43.183702+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:43.183702+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9619, 'log_dups_size': 0, 'ondisk_log_size': 9619, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20426480, 'num_objects': 2494, 'num_object_clones': 0, 'num_object_copies': 9976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9616, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 638}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9703", 'reported_seq': 10005, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:43.289446+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:43.289446+0000', 'last_peered': '2024-09-10T20:27:43.289446+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:43.289446+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9703, 'log_dups_size': 0, 'ondisk_log_size': 9703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20167248, 'num_objects': 2462, 'num_object_clones': 0, 'num_object_copies': 9848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9702, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 602}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9437", 'reported_seq': 9624, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:44.770816+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:44.770816+0000', 'last_peered': '2024-09-10T20:27:44.770816+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:44.770816+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9437, 'log_dups_size': 0, 'ondisk_log_size': 9437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19610192, 'num_objects': 2394, 'num_object_clones': 0, 'num_object_copies': 9576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9436, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:46.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:46.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9675", 'reported_seq': 10769, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:43.324422+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:43.324422+0000', 'last_peered': '2024-09-10T20:27:43.324422+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:43.324422+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9675, 'log_dups_size': 0, 'ondisk_log_size': 9675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19813536, 'num_objects': 2419, 'num_object_clones': 0, 'num_object_copies': 9676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1467, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9673, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 512, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:46.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:46.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9927", 'reported_seq': 10136, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:44.876859+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:44.876859+0000', 'last_peered': '2024-09-10T20:27:44.876859+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:44.876859+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9927, 'log_dups_size': 0, 'ondisk_log_size': 9927, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20657312, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9925, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:46.297 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:47.054 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:47.055 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20376/158604 objects degraded (12.847%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":555,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39651,"data_bytes":324788072,"bytes_used":2753916928,"bytes_avail":383793139712,"bytes_total":386547056640,"degraded_objects":20376,"degraded_total":158604,"degraded_ratio":0.12847090867821745,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":205},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-10T20:27:40.057375+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:27:47.080 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:47.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:47.698 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:47.698 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:47.734 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-10T20:27:46.059217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":324788072,"num_objects":39651,"num_object_clones":0,"num_object_copies":158604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125567,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154818,"ondisk_log_size":154818,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2689372,"kb_used_data":639004,"kb_used_omap":14,"kb_used_meta":2050289,"kb_avail":374797988,"statfs":{"total":386547056640,"available":383793139712,"internally_reserved":0,"allocated":654340096,"data_stored":653450032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2099496502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,10,0,5,9,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20168704,"num_objects":-2462,"num_object_clones":0,"num_object_copies":-9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2462,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003968"},"pg_stats":[{"pgid":"1.f","version":"42'9650","reported_seq":9933,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.313689+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:43.313689+0000","last_peered":"2024-09-10T20:27:43.313689+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:43.313689+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9712","reported_seq":9848,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.074400+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:41.074400+0000","last_peered":"2024-09-10T20:27:41.074400+0000","last_clean":"2024-09-10T20:27:41.074400+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:41.074400+0000","last_undegraded":"2024-09-10T20:27:41.074400+0000","last_fullsized":"2024-09-10T20:27:41.074400+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9712,"log_dups_size":0,"ondisk_log_size":9712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20807680,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9667","reported_seq":9872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:45.022504+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:45.022504+0000","last_peered":"2024-09-10T20:27:45.022504+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:45.022504+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20257384,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9666,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9878","reported_seq":14052,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.064951+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:41.064951+0000","last_peered":"2024-09-10T20:27:41.064951+0000","last_clean":"2024-09-10T20:27:41.064951+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:41.064951+0000","last_undegraded":"2024-09-10T20:27:41.064951+0000","last_fullsized":"2024-09-10T20:27:41.064951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9718","reported_seq":10041,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.365575+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:43.365575+0000","last_peered":"2024-09-10T20:27:43.365575+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:43.365575+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9718,"log_dups_size":0,"ondisk_log_size":9718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075680,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9668","reported_seq":9894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.278012+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:43.278012+0000","last_peered":"2024-09-10T20:27:43.278012+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:43.278012+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19906560,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9631","reported_seq":9839,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:42.173275+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:42.173275+0000","last_peered":"2024-09-10T20:27:42.173275+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:42.173275+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19977376,"num_objects":2439,"num_object_clones":0,"num_object_copies":9756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":546}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9787","reported_seq":9958,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:42.133919+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:42.133919+0000","last_peered":"2024-09-10T20:27:42.133919+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:42.133919+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9787,"log_dups_size":0,"ondisk_log_size":9787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20526240,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9532","reported_seq":13508,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:43.365510+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:43.365510+0000","last_peered":"2024-09-10T20:27:43.365510+0000","last_clean":"2024-09-10T20:27:43.365510+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:43.365510+0000","last_undegraded":"2024-09-10T20:27:43.365510+0000","last_fullsized":"2024-09-10T20:27:43.365510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9532,"log_dups_size":0,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20224592,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9609","reported_seq":13584,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:44.986094+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:44.986094+0000","last_peered":"2024-09-10T20:27:44.986094+0000","last_clean":"2024-09-10T20:27:44.986094+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:44.986094+0000","last_undegraded":"2024-09-10T20:27:44.986094+0000","last_fullsized":"2024-09-10T20:27:44.986094+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20750336,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9619","reported_seq":9896,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.183702+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:43.183702+0000","last_peered":"2024-09-10T20:27:43.183702+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:43.183702+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9619,"log_dups_size":0,"ondisk_log_size":9619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20426480,"num_objects":2494,"num_object_clones":0,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9703","reported_seq":10005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.289446+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:43.289446+0000","last_peered":"2024-09-10T20:27:43.289446+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:43.289446+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20167248,"num_objects":2462,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":602}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9437","reported_seq":9624,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.770816+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:44.770816+0000","last_peered":"2024-09-10T20:27:44.770816+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:44.770816+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9437,"log_dups_size":0,"ondisk_log_size":9437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19610192,"num_objects":2394,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9675","reported_seq":10769,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:43.324422+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:43.324422+0000","last_peered":"2024-09-10T20:27:43.324422+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:43.324422+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19813536,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9673,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9927","reported_seq":10136,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.876859+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:44.876859+0000","last_peered":"2024-09-10T20:27:44.876859+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:44.876859+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9605","reported_seq":13705,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:41.133771+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:41.133771+0000","last_peered":"2024-09-10T20:27:41.133771+0000","last_clean":"2024-09-10T20:27:41.133771+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:41.133771+0000","last_undegraded":"2024-09-10T20:27:41.133771+0000","last_fullsized":"2024-09-10T20:27:41.133771+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":478,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":324788072,"num_objects":39651,"num_object_clones":0,"num_object_copies":158604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125567,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":650784768,"data_stored":650784768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154818,"ondisk_log_size":154818,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117096,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704940,"kb_used_data":159580,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93666900,"statfs":{"total":96636764160,"available":95914905600,"internally_reserved":0,"allocated":163409920,"data_stored":163187404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694848,"kb_used_data":158320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93676992,"statfs":{"total":96636764160,"available":95925239808,"internally_reserved":0,"allocated":162119680,"data_stored":161897164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598024,"kb_used_data":160952,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93773816,"statfs":{"total":96636764160,"available":96024387584,"internally_reserved":0,"allocated":164814848,"data_stored":164592332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447540946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691560,"kb_used_data":160152,"kb_used_omap":4,"kb_used_meta":531387,"kb_avail":93680280,"statfs":{"total":96636764160,"available":95928606720,"internally_reserved":0,"allocated":163995648,"data_stored":163773132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":544141007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163397632,"data_stored":163397632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164306944,"data_stored":164306944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160813056,"data_stored":160813056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":162267136,"data_stored":162267136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:50.736 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:51.368 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:51.369 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:51.393 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-10T20:27:50.060796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318070632,"num_objects":38831,"num_object_clones":0,"num_object_copies":155324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126387,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155638,"ondisk_log_size":155638,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2696832,"kb_used_data":625572,"kb_used_omap":14,"kb_used_meta":2071153,"kb_avail":374790528,"statfs":{"total":386547056640,"available":383785500672,"internally_reserved":0,"allocated":640585728,"data_stored":639695664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2120861238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,7,6,15,11,15,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21766144,"num_objects":-2657,"num_object_clones":0,"num_object_copies":-10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004052"},"pg_stats":[{"pgid":"1.f","version":"42'9708","reported_seq":9994,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.047464+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:48.047464+0000","last_peered":"2024-09-10T20:27:48.047464+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:48.047464+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9708,"log_dups_size":0,"ondisk_log_size":9708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775488,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9779","reported_seq":9915,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:46.170635+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:46.170635+0000","last_peered":"2024-09-10T20:27:46.170635+0000","last_clean":"2024-09-10T20:27:46.170635+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:46.170635+0000","last_undegraded":"2024-09-10T20:27:46.170635+0000","last_fullsized":"2024-09-10T20:27:46.170635+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9779,"log_dups_size":0,"ondisk_log_size":9779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20258816,"num_objects":2473,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6220,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9667","reported_seq":9872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:45.022504+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:45.022504+0000","last_peered":"2024-09-10T20:27:45.022504+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:45.022504+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20257384,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9666,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9937","reported_seq":14111,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:46.183398+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:46.183398+0000","last_peered":"2024-09-10T20:27:46.183398+0000","last_clean":"2024-09-10T20:27:46.183398+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:46.183398+0000","last_undegraded":"2024-09-10T20:27:46.183398+0000","last_fullsized":"2024-09-10T20:27:46.183398+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20226048,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9784","reported_seq":10109,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.370539+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:48.370539+0000","last_peered":"2024-09-10T20:27:48.370539+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:48.370539+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9784,"log_dups_size":0,"ondisk_log_size":9784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19535008,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9758","reported_seq":9984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.494093+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:48.494093+0000","last_peered":"2024-09-10T20:27:48.494093+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:48.494093+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19169280,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9701","reported_seq":9908,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:47.192541+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:47.192541+0000","last_peered":"2024-09-10T20:27:47.192541+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:47.192541+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19403936,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":476}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9849","reported_seq":10022,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:47.229137+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:47.229137+0000","last_peered":"2024-09-10T20:27:47.229137+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:47.229137+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018336,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9847,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9600","reported_seq":13574,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:48.436639+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:48.436639+0000","last_peered":"2024-09-10T20:27:48.436639+0000","last_clean":"2024-09-10T20:27:48.436639+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:48.436639+0000","last_undegraded":"2024-09-10T20:27:48.436639+0000","last_fullsized":"2024-09-10T20:27:48.436639+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9600,"log_dups_size":0,"ondisk_log_size":9600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19667536,"num_objects":2401,"num_object_clones":0,"num_object_copies":9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9609","reported_seq":13584,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:44.986094+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:44.986094+0000","last_peered":"2024-09-10T20:27:44.986094+0000","last_clean":"2024-09-10T20:27:44.986094+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:44.986094+0000","last_undegraded":"2024-09-10T20:27:44.986094+0000","last_fullsized":"2024-09-10T20:27:44.986094+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20750336,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9693","reported_seq":9969,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.482761+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:48.482761+0000","last_peered":"2024-09-10T20:27:48.482761+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:48.482761+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19820272,"num_objects":2420,"num_object_clones":0,"num_object_copies":9680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9783","reported_seq":10085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.435611+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:48.435611+0000","last_peered":"2024-09-10T20:27:48.435611+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:48.435611+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511888,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9437","reported_seq":9624,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.770816+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:44.770816+0000","last_peered":"2024-09-10T20:27:44.770816+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:44.770816+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9437,"log_dups_size":0,"ondisk_log_size":9437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19610192,"num_objects":2394,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9731","reported_seq":10911,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.349167+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:48.349167+0000","last_peered":"2024-09-10T20:27:48.349167+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:48.349167+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19354784,"num_objects":2363,"num_object_clones":0,"num_object_copies":9452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9729,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9927","reported_seq":10136,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.876859+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:44.876859+0000","last_peered":"2024-09-10T20:27:44.876859+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:44.876859+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9675","reported_seq":13775,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:46.278644+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:46.278644+0000","last_peered":"2024-09-10T20:27:46.278644+0000","last_clean":"2024-09-10T20:27:46.278644+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:46.278644+0000","last_undegraded":"2024-09-10T20:27:46.278644+0000","last_fullsized":"2024-09-10T20:27:46.278644+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":318070632,"num_objects":38831,"num_object_clones":0,"num_object_copies":155324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126387,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":637489152,"data_stored":637489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155638,"ondisk_log_size":155638,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727768,"kb_used_data":154628,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93644072,"statfs":{"total":96636764160,"available":95891529728,"internally_reserved":0,"allocated":158339072,"data_stored":158116556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,9,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694848,"kb_used_data":158320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93676992,"statfs":{"total":96636764160,"available":95925239808,"internally_reserved":0,"allocated":162119680,"data_stored":161897164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581320,"kb_used_data":156708,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93790520,"statfs":{"total":96636764160,"available":96041492480,"internally_reserved":0,"allocated":160468992,"data_stored":160246476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":434761426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,0,2,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692896,"kb_used_data":155916,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93678944,"statfs":{"total":96636764160,"available":95927238656,"internally_reserved":0,"allocated":159657984,"data_stored":159435468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158994432,"data_stored":158994432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159805440,"data_stored":159805440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160813056,"data_stored":160813056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157876224,"data_stored":157876224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:51.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:52.020 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:52.020 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:52.039 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-10T20:27:50.060796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318070632,"num_objects":38831,"num_object_clones":0,"num_object_copies":155324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126387,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155638,"ondisk_log_size":155638,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2696832,"kb_used_data":625572,"kb_used_omap":14,"kb_used_meta":2071153,"kb_avail":374790528,"statfs":{"total":386547056640,"available":383785500672,"internally_reserved":0,"allocated":640585728,"data_stored":639695664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2120861238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,7,6,15,11,15,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21766144,"num_objects":-2657,"num_object_clones":0,"num_object_copies":-10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004052"},"pg_stats":[{"pgid":"1.f","version":"42'9708","reported_seq":9994,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.047464+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:48.047464+0000","last_peered":"2024-09-10T20:27:48.047464+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:48.047464+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9708,"log_dups_size":0,"ondisk_log_size":9708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775488,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9779","reported_seq":9915,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:46.170635+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:46.170635+0000","last_peered":"2024-09-10T20:27:46.170635+0000","last_clean":"2024-09-10T20:27:46.170635+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:46.170635+0000","last_undegraded":"2024-09-10T20:27:46.170635+0000","last_fullsized":"2024-09-10T20:27:46.170635+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9779,"log_dups_size":0,"ondisk_log_size":9779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20258816,"num_objects":2473,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6220,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9667","reported_seq":9872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:45.022504+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:45.022504+0000","last_peered":"2024-09-10T20:27:45.022504+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:45.022504+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20257384,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9666,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9937","reported_seq":14111,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:46.183398+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:46.183398+0000","last_peered":"2024-09-10T20:27:46.183398+0000","last_clean":"2024-09-10T20:27:46.183398+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:46.183398+0000","last_undegraded":"2024-09-10T20:27:46.183398+0000","last_fullsized":"2024-09-10T20:27:46.183398+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20226048,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9784","reported_seq":10109,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.370539+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:48.370539+0000","last_peered":"2024-09-10T20:27:48.370539+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:48.370539+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9784,"log_dups_size":0,"ondisk_log_size":9784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19535008,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9758","reported_seq":9984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.494093+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:48.494093+0000","last_peered":"2024-09-10T20:27:48.494093+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:48.494093+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19169280,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9701","reported_seq":9908,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:47.192541+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:47.192541+0000","last_peered":"2024-09-10T20:27:47.192541+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:47.192541+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19403936,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":476}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9849","reported_seq":10022,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:47.229137+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:47.229137+0000","last_peered":"2024-09-10T20:27:47.229137+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:47.229137+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018336,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9847,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9600","reported_seq":13574,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:48.436639+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:48.436639+0000","last_peered":"2024-09-10T20:27:48.436639+0000","last_clean":"2024-09-10T20:27:48.436639+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:48.436639+0000","last_undegraded":"2024-09-10T20:27:48.436639+0000","last_fullsized":"2024-09-10T20:27:48.436639+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9600,"log_dups_size":0,"ondisk_log_size":9600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19667536,"num_objects":2401,"num_object_clones":0,"num_object_copies":9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9609","reported_seq":13584,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:44.986094+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:44.986094+0000","last_peered":"2024-09-10T20:27:44.986094+0000","last_clean":"2024-09-10T20:27:44.986094+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:44.986094+0000","last_undegraded":"2024-09-10T20:27:44.986094+0000","last_fullsized":"2024-09-10T20:27:44.986094+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20750336,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9693","reported_seq":9969,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.482761+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:48.482761+0000","last_peered":"2024-09-10T20:27:48.482761+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:48.482761+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19820272,"num_objects":2420,"num_object_clones":0,"num_object_copies":9680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9783","reported_seq":10085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.435611+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:48.435611+0000","last_peered":"2024-09-10T20:27:48.435611+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:48.435611+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511888,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9437","reported_seq":9624,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.770816+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:44.770816+0000","last_peered":"2024-09-10T20:27:44.770816+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:44.770816+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9437,"log_dups_size":0,"ondisk_log_size":9437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19610192,"num_objects":2394,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9731","reported_seq":10911,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.349167+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:48.349167+0000","last_peered":"2024-09-10T20:27:48.349167+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:48.349167+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19354784,"num_objects":2363,"num_object_clones":0,"num_object_copies":9452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9729,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9927","reported_seq":10136,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:44.876859+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:44.876859+0000","last_peered":"2024-09-10T20:27:44.876859+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:44.876859+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9675","reported_seq":13775,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:46.278644+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:46.278644+0000","last_peered":"2024-09-10T20:27:46.278644+0000","last_clean":"2024-09-10T20:27:46.278644+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:46.278644+0000","last_undegraded":"2024-09-10T20:27:46.278644+0000","last_fullsized":"2024-09-10T20:27:46.278644+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":318070632,"num_objects":38831,"num_object_clones":0,"num_object_copies":155324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126387,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":637489152,"data_stored":637489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155638,"ondisk_log_size":155638,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727768,"kb_used_data":154628,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93644072,"statfs":{"total":96636764160,"available":95891529728,"internally_reserved":0,"allocated":158339072,"data_stored":158116556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,9,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694848,"kb_used_data":158320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93676992,"statfs":{"total":96636764160,"available":95925239808,"internally_reserved":0,"allocated":162119680,"data_stored":161897164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581320,"kb_used_data":156708,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93790520,"statfs":{"total":96636764160,"available":96041492480,"internally_reserved":0,"allocated":160468992,"data_stored":160246476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":434761426},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,0,2,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692896,"kb_used_data":155916,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93678944,"statfs":{"total":96636764160,"available":95927238656,"internally_reserved":0,"allocated":159657984,"data_stored":159435468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158994432,"data_stored":158994432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159805440,"data_stored":159805440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160813056,"data_stored":160813056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157876224,"data_stored":157876224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:52.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:52.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9708", 'reported_seq': 9994, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:48.047464+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:48.047464+0000', 'last_peered': '2024-09-10T20:27:48.047464+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:48.047464+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9708, 'log_dups_size': 0, 'ondisk_log_size': 9708, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19775488, 'num_objects': 2414, 'num_object_clones': 0, 'num_object_copies': 9656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9708, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 559}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:52.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:52.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9667", 'reported_seq': 9872, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:45.022504+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:45.022504+0000', 'last_peered': '2024-09-10T20:27:45.022504+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:45.022504+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9667, 'log_dups_size': 0, 'ondisk_log_size': 9667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20257384, 'num_objects': 2474, 'num_object_clones': 0, 'num_object_copies': 9896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2474, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9666, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 566}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:52.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:52.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9784", 'reported_seq': 10109, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:48.370539+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:48.370539+0000', 'last_peered': '2024-09-10T20:27:48.370539+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:48.370539+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9784, 'log_dups_size': 0, 'ondisk_log_size': 9784, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19535008, 'num_objects': 2385, 'num_object_clones': 0, 'num_object_copies': 9540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9782, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 530}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9758", 'reported_seq': 9984, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:48.494093+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:48.494093+0000', 'last_peered': '2024-09-10T20:27:48.494093+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:48.494093+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9758, 'log_dups_size': 0, 'ondisk_log_size': 9758, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19169280, 'num_objects': 2340, 'num_object_clones': 0, 'num_object_copies': 9360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9758, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 483}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9701", 'reported_seq': 9908, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:47.192541+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:47.192541+0000', 'last_peered': '2024-09-10T20:27:47.192541+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:47.192541+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9701, 'log_dups_size': 0, 'ondisk_log_size': 9701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19403936, 'num_objects': 2369, 'num_object_clones': 0, 'num_object_copies': 9476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9699, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 476}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9849", 'reported_seq': 10022, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:47.229137+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:47.229137+0000', 'last_peered': '2024-09-10T20:27:47.229137+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:47.229137+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9849, 'log_dups_size': 0, 'ondisk_log_size': 9849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20018336, 'num_objects': 2444, 'num_object_clones': 0, 'num_object_copies': 9776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9847, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9693", 'reported_seq': 9969, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:48.482761+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:48.482761+0000', 'last_peered': '2024-09-10T20:27:48.482761+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:48.482761+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9693, 'log_dups_size': 0, 'ondisk_log_size': 9693, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19820272, 'num_objects': 2420, 'num_object_clones': 0, 'num_object_copies': 9680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9690, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 564}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9783", 'reported_seq': 10085, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:48.435611+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:48.435611+0000', 'last_peered': '2024-09-10T20:27:48.435611+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:48.435611+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9783, 'log_dups_size': 0, 'ondisk_log_size': 9783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19511888, 'num_objects': 2382, 'num_object_clones': 0, 'num_object_copies': 9528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2382, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9782, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:52.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9437", 'reported_seq': 9624, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:44.770816+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:44.770816+0000', 'last_peered': '2024-09-10T20:27:44.770816+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:44.770816+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9437, 'log_dups_size': 0, 'ondisk_log_size': 9437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19610192, 'num_objects': 2394, 'num_object_clones': 0, 'num_object_copies': 9576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9436, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:52.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:52.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9731", 'reported_seq': 10911, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:48.349167+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:48.349167+0000', 'last_peered': '2024-09-10T20:27:48.349167+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:48.349167+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9731, 'log_dups_size': 0, 'ondisk_log_size': 9731, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19354784, 'num_objects': 2363, 'num_object_clones': 0, 'num_object_copies': 9452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1424, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9729, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 555, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:52.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:52.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'9927", 'reported_seq': 10136, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:44.876859+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:44.876859+0000', 'last_peered': '2024-09-10T20:27:44.876859+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:44.876859+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9927, 'log_dups_size': 0, 'ondisk_log_size': 9927, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20657312, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9925, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:52.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:52.754 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:52.754 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20333/153376 objects degraded (13.257%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":561,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38344,"data_bytes":314081128,"bytes_used":2815815680,"bytes_avail":383731240960,"bytes_total":386547056640,"degraded_objects":20333,"degraded_total":153376,"degraded_ratio":0.13256963279783016,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-10T20:27:40.057375+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:27:52.773 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:52.773 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:53.343 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-10T20:27:52.061470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":314081128,"num_objects":38344,"num_object_clones":0,"num_object_copies":153376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126874,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156125,"ondisk_log_size":156125,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749820,"kb_used_data":616104,"kb_used_omap":14,"kb_used_meta":2133617,"kb_avail":374737540,"statfs":{"total":386547056640,"available":383731240960,"internally_reserved":0,"allocated":630890496,"data_stored":630000432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2184824374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,6,7,17,23,23,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19562496,"num_objects":-2388,"num_object_clones":0,"num_object_copies":-9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2388,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004190"},"pg_stats":[{"pgid":"1.f","version":"42'9708","reported_seq":9994,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.047464+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:48.047464+0000","last_peered":"2024-09-10T20:27:48.047464+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:48.047464+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9708,"log_dups_size":0,"ondisk_log_size":9708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775488,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9860","reported_seq":9994,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.405309+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:51.405309+0000","last_peered":"2024-09-10T20:27:51.405309+0000","last_clean":"2024-09-10T20:27:51.405309+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:51.405309+0000","last_undegraded":"2024-09-10T20:27:51.405309+0000","last_fullsized":"2024-09-10T20:27:51.405309+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9860,"log_dups_size":0,"ondisk_log_size":9860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6301,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9742","reported_seq":9949,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:49.933565+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:49.933565+0000","last_peered":"2024-09-10T20:27:49.933565+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:49.933565+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19642984,"num_objects":2399,"num_object_clones":0,"num_object_copies":9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9741,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9999","reported_seq":14172,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.345743+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:51.345743+0000","last_peered":"2024-09-10T20:27:51.345743+0000","last_clean":"2024-09-10T20:27:51.345743+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:51.345743+0000","last_undegraded":"2024-09-10T20:27:51.345743+0000","last_fullsized":"2024-09-10T20:27:51.345743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9784","reported_seq":10109,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.370539+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:48.370539+0000","last_peered":"2024-09-10T20:27:48.370539+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:48.370539+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9784,"log_dups_size":0,"ondisk_log_size":9784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19535008,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9758","reported_seq":9984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.494093+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:48.494093+0000","last_peered":"2024-09-10T20:27:48.494093+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:48.494093+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19169280,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9701","reported_seq":9908,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:47.192541+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:47.192541+0000","last_peered":"2024-09-10T20:27:47.192541+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:47.192541+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19403936,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":476}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9849","reported_seq":10022,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:47.229137+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:47.229137+0000","last_peered":"2024-09-10T20:27:47.229137+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:47.229137+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018336,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9847,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9600","reported_seq":13574,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:48.436639+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:48.436639+0000","last_peered":"2024-09-10T20:27:48.436639+0000","last_clean":"2024-09-10T20:27:48.436639+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:48.436639+0000","last_undegraded":"2024-09-10T20:27:48.436639+0000","last_fullsized":"2024-09-10T20:27:48.436639+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9600,"log_dups_size":0,"ondisk_log_size":9600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19667536,"num_objects":2401,"num_object_clones":0,"num_object_copies":9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9682","reported_seq":13657,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:50.130573+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:50.130573+0000","last_peered":"2024-09-10T20:27:50.130573+0000","last_clean":"2024-09-10T20:27:50.130573+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:50.130573+0000","last_undegraded":"2024-09-10T20:27:50.130573+0000","last_fullsized":"2024-09-10T20:27:50.130573+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9682,"log_dups_size":0,"ondisk_log_size":9682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9693","reported_seq":9969,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.482761+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:48.482761+0000","last_peered":"2024-09-10T20:27:48.482761+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:48.482761+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19820272,"num_objects":2420,"num_object_clones":0,"num_object_copies":9680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9783","reported_seq":10085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.435611+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:48.435611+0000","last_peered":"2024-09-10T20:27:48.435611+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:48.435611+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511888,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9496","reported_seq":9683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:50.122883+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:50.122883+0000","last_peered":"2024-09-10T20:27:50.122883+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:50.122883+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19126864,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9495,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9731","reported_seq":10911,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:48.349167+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:48.349167+0000","last_peered":"2024-09-10T20:27:48.349167+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:48.349167+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19354784,"num_objects":2363,"num_object_clones":0,"num_object_copies":9452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9729,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'9988","reported_seq":10194,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:49.977714+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:49.977714+0000","last_peered":"2024-09-10T20:27:49.977714+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:49.977714+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9988,"log_dups_size":0,"ondisk_log_size":9988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20157600,"num_objects":2461,"num_object_clones":0,"num_object_copies":9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9986,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9751","reported_seq":13851,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.275462+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:51.275462+0000","last_peered":"2024-09-10T20:27:51.275462+0000","last_clean":"2024-09-10T20:27:51.275462+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:51.275462+0000","last_undegraded":"2024-09-10T20:27:51.275462+0000","last_fullsized":"2024-09-10T20:27:51.275462+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19431424,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":314081128,"num_objects":38344,"num_object_clones":0,"num_object_copies":153376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126874,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":628482048,"data_stored":628482048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156125,"ondisk_log_size":156125,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727768,"kb_used_data":154628,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93644072,"statfs":{"total":96636764160,"available":95891529728,"internally_reserved":0,"allocated":158339072,"data_stored":158116556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,9,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729920,"kb_used_data":153124,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93641920,"statfs":{"total":96636764160,"available":95889326080,"internally_reserved":0,"allocated":156798976,"data_stored":156576460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,0,3,2,5,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599236,"kb_used_data":152436,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93772604,"statfs":{"total":96636764160,"available":96023146496,"internally_reserved":0,"allocated":156094464,"data_stored":155871948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692896,"kb_used_data":155916,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93678944,"statfs":{"total":96636764160,"available":95927238656,"internally_reserved":0,"allocated":159657984,"data_stored":159435468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158994432,"data_stored":158994432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155250688,"data_stored":155250688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":156360704,"data_stored":156360704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157876224,"data_stored":157876224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:56.370 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:56.950 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:56.951 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:56.975 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-10T20:27:56.062773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":307019624,"num_objects":37482,"num_object_clones":0,"num_object_copies":149928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37482,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127736,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156987,"ondisk_log_size":156987,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818656,"kb_used_data":604424,"kb_used_omap":14,"kb_used_meta":2214129,"kb_avail":374668704,"statfs":{"total":386547056640,"available":383660752896,"internally_reserved":0,"allocated":618930176,"data_stored":618040112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2267268662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,9,6,23,15,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20078592,"num_objects":-2451,"num_object_clones":0,"num_object_copies":-9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004238"},"pg_stats":[{"pgid":"1.f","version":"42'9785","reported_seq":10071,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.451056+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:53.451056+0000","last_peered":"2024-09-10T20:27:53.451056+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:53.451056+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9785,"log_dups_size":0,"ondisk_log_size":9785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19144704,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9860","reported_seq":9994,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.405309+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:51.405309+0000","last_peered":"2024-09-10T20:27:51.405309+0000","last_clean":"2024-09-10T20:27:51.405309+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:51.405309+0000","last_undegraded":"2024-09-10T20:27:51.405309+0000","last_fullsized":"2024-09-10T20:27:51.405309+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9860,"log_dups_size":0,"ondisk_log_size":9860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6301,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9814","reported_seq":10021,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.054436+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:55.054436+0000","last_peered":"2024-09-10T20:27:55.054436+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:55.054436+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19053160,"num_objects":2327,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9813,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9999","reported_seq":14172,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.345743+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:51.345743+0000","last_peered":"2024-09-10T20:27:51.345743+0000","last_clean":"2024-09-10T20:27:51.345743+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:51.345743+0000","last_undegraded":"2024-09-10T20:27:51.345743+0000","last_fullsized":"2024-09-10T20:27:51.345743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9845","reported_seq":10170,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.080320+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:53.080320+0000","last_peered":"2024-09-10T20:27:53.080320+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:53.080320+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9845,"log_dups_size":0,"ondisk_log_size":9845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19035296,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9843,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9811","reported_seq":10037,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.132224+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:53.132224+0000","last_peered":"2024-09-10T20:27:53.132224+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:53.132224+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18735104,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9774","reported_seq":9982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:52.330588+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:52.330588+0000","last_peered":"2024-09-10T20:27:52.330588+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:52.330588+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18805920,"num_objects":2296,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9927","reported_seq":10100,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:52.152785+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:52.152785+0000","last_peered":"2024-09-10T20:27:52.152785+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:52.152785+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379360,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9670","reported_seq":13645,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:53.545273+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:53.545273+0000","last_peered":"2024-09-10T20:27:53.545273+0000","last_clean":"2024-09-10T20:27:53.545273+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:53.545273+0000","last_undegraded":"2024-09-10T20:27:53.545273+0000","last_fullsized":"2024-09-10T20:27:53.545273+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19094096,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6223,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9754","reported_seq":13728,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:55.064787+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:55.064787+0000","last_peered":"2024-09-10T20:27:55.064787+0000","last_clean":"2024-09-10T20:27:55.064787+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:55.064787+0000","last_undegraded":"2024-09-10T20:27:55.064787+0000","last_fullsized":"2024-09-10T20:27:55.064787+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19562496,"num_objects":2388,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9748","reported_seq":10025,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.461903+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:53.461903+0000","last_peered":"2024-09-10T20:27:53.461903+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:53.461903+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19369712,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9837","reported_seq":10138,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.474615+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:53.474615+0000","last_peered":"2024-09-10T20:27:53.474615+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:53.474615+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9837,"log_dups_size":0,"ondisk_log_size":9837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19069520,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9563","reported_seq":9749,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.151697+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:55.151697+0000","last_peered":"2024-09-10T20:27:55.151697+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:55.151697+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18578000,"num_objects":2268,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":365}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9786","reported_seq":11066,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.478635+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:53.478635+0000","last_peered":"2024-09-10T20:27:53.478635+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:53.478635+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9786,"log_dups_size":0,"ondisk_log_size":9786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904224,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9784,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10063","reported_seq":10273,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.137244+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:55.137244+0000","last_peered":"2024-09-10T20:27:55.137244+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:55.137244+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19543200,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10061,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9751","reported_seq":13851,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.275462+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:51.275462+0000","last_peered":"2024-09-10T20:27:51.275462+0000","last_clean":"2024-09-10T20:27:51.275462+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:51.275462+0000","last_undegraded":"2024-09-10T20:27:51.275462+0000","last_fullsized":"2024-09-10T20:27:51.275462+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19431424,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":307019624,"num_objects":37482,"num_object_clones":0,"num_object_copies":149928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37482,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127736,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":615071744,"data_stored":615071744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156987,"ondisk_log_size":156987,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117098,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724072,"kb_used_data":150936,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93647768,"statfs":{"total":96636764160,"available":95895314432,"internally_reserved":0,"allocated":154558464,"data_stored":154335948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,6,3,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748516,"kb_used_data":149508,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93623324,"statfs":{"total":96636764160,"available":95870283776,"internally_reserved":0,"allocated":153096192,"data_stored":152873676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,10,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599236,"kb_used_data":152436,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93772604,"statfs":{"total":96636764160,"available":96023146496,"internally_reserved":0,"allocated":156094464,"data_stored":155871948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746832,"kb_used_data":151544,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93625008,"statfs":{"total":96636764160,"available":95872008192,"internally_reserved":0,"allocated":155181056,"data_stored":154958540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":154415104,"data_stored":154415104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155250688,"data_stored":155250688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151924736,"data_stored":151924736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153481216,"data_stored":153481216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:56.976 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:57.612 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:57.613 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:57.631 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-10T20:27:56.062773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":307019624,"num_objects":37482,"num_object_clones":0,"num_object_copies":149928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37482,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127736,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156987,"ondisk_log_size":156987,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818656,"kb_used_data":604424,"kb_used_omap":14,"kb_used_meta":2214129,"kb_avail":374668704,"statfs":{"total":386547056640,"available":383660752896,"internally_reserved":0,"allocated":618930176,"data_stored":618040112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2267268662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,9,6,23,15,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20078592,"num_objects":-2451,"num_object_clones":0,"num_object_copies":-9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004238"},"pg_stats":[{"pgid":"1.f","version":"42'9785","reported_seq":10071,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.451056+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:53.451056+0000","last_peered":"2024-09-10T20:27:53.451056+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:53.451056+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9785,"log_dups_size":0,"ondisk_log_size":9785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19144704,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9860","reported_seq":9994,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.405309+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:51.405309+0000","last_peered":"2024-09-10T20:27:51.405309+0000","last_clean":"2024-09-10T20:27:51.405309+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:51.405309+0000","last_undegraded":"2024-09-10T20:27:51.405309+0000","last_fullsized":"2024-09-10T20:27:51.405309+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9860,"log_dups_size":0,"ondisk_log_size":9860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6301,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9814","reported_seq":10021,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.054436+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:55.054436+0000","last_peered":"2024-09-10T20:27:55.054436+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:55.054436+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19053160,"num_objects":2327,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9813,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'9999","reported_seq":14172,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.345743+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:51.345743+0000","last_peered":"2024-09-10T20:27:51.345743+0000","last_clean":"2024-09-10T20:27:51.345743+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:51.345743+0000","last_undegraded":"2024-09-10T20:27:51.345743+0000","last_fullsized":"2024-09-10T20:27:51.345743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9845","reported_seq":10170,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.080320+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:53.080320+0000","last_peered":"2024-09-10T20:27:53.080320+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:53.080320+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9845,"log_dups_size":0,"ondisk_log_size":9845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19035296,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9843,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9811","reported_seq":10037,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.132224+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:53.132224+0000","last_peered":"2024-09-10T20:27:53.132224+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:53.132224+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18735104,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9774","reported_seq":9982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:52.330588+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:52.330588+0000","last_peered":"2024-09-10T20:27:52.330588+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:52.330588+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18805920,"num_objects":2296,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9927","reported_seq":10100,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:52.152785+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:52.152785+0000","last_peered":"2024-09-10T20:27:52.152785+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:52.152785+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379360,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9670","reported_seq":13645,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:53.545273+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:53.545273+0000","last_peered":"2024-09-10T20:27:53.545273+0000","last_clean":"2024-09-10T20:27:53.545273+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:53.545273+0000","last_undegraded":"2024-09-10T20:27:53.545273+0000","last_fullsized":"2024-09-10T20:27:53.545273+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19094096,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6223,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9754","reported_seq":13728,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:55.064787+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:55.064787+0000","last_peered":"2024-09-10T20:27:55.064787+0000","last_clean":"2024-09-10T20:27:55.064787+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:55.064787+0000","last_undegraded":"2024-09-10T20:27:55.064787+0000","last_fullsized":"2024-09-10T20:27:55.064787+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19562496,"num_objects":2388,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9748","reported_seq":10025,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.461903+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:53.461903+0000","last_peered":"2024-09-10T20:27:53.461903+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:53.461903+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19369712,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9837","reported_seq":10138,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.474615+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:53.474615+0000","last_peered":"2024-09-10T20:27:53.474615+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:53.474615+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9837,"log_dups_size":0,"ondisk_log_size":9837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19069520,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9563","reported_seq":9749,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.151697+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:55.151697+0000","last_peered":"2024-09-10T20:27:55.151697+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:55.151697+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18578000,"num_objects":2268,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":365}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9786","reported_seq":11066,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.478635+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:53.478635+0000","last_peered":"2024-09-10T20:27:53.478635+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:53.478635+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9786,"log_dups_size":0,"ondisk_log_size":9786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904224,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9784,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10063","reported_seq":10273,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.137244+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:55.137244+0000","last_peered":"2024-09-10T20:27:55.137244+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:55.137244+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19543200,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10061,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9751","reported_seq":13851,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:51.275462+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:51.275462+0000","last_peered":"2024-09-10T20:27:51.275462+0000","last_clean":"2024-09-10T20:27:51.275462+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:51.275462+0000","last_undegraded":"2024-09-10T20:27:51.275462+0000","last_fullsized":"2024-09-10T20:27:51.275462+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19431424,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":307019624,"num_objects":37482,"num_object_clones":0,"num_object_copies":149928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37482,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127736,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":615071744,"data_stored":615071744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156987,"ondisk_log_size":156987,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117098,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724072,"kb_used_data":150936,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93647768,"statfs":{"total":96636764160,"available":95895314432,"internally_reserved":0,"allocated":154558464,"data_stored":154335948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,6,3,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748516,"kb_used_data":149508,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93623324,"statfs":{"total":96636764160,"available":95870283776,"internally_reserved":0,"allocated":153096192,"data_stored":152873676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,10,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599236,"kb_used_data":152436,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93772604,"statfs":{"total":96636764160,"available":96023146496,"internally_reserved":0,"allocated":156094464,"data_stored":155871948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746832,"kb_used_data":151544,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93625008,"statfs":{"total":96636764160,"available":95872008192,"internally_reserved":0,"allocated":155181056,"data_stored":154958540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":154415104,"data_stored":154415104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155250688,"data_stored":155250688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151924736,"data_stored":151924736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153481216,"data_stored":153481216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:27:57.632 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:27:57.632 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9785", 'reported_seq': 10071, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:53.451056+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:53.451056+0000', 'last_peered': '2024-09-10T20:27:53.451056+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:53.451056+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9785, 'log_dups_size': 0, 'ondisk_log_size': 9785, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19144704, 'num_objects': 2337, 'num_object_clones': 0, 'num_object_copies': 9348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9785, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 482}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:57.632 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9814", 'reported_seq': 10021, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:55.054436+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:27:55.054436+0000', 'last_peered': '2024-09-10T20:27:55.054436+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:27:55.054436+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9814, 'log_dups_size': 0, 'ondisk_log_size': 9814, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19053160, 'num_objects': 2327, 'num_object_clones': 0, 'num_object_copies': 9308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2327, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9813, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 419}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9845", 'reported_seq': 10170, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:53.080320+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:53.080320+0000', 'last_peered': '2024-09-10T20:27:53.080320+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:53.080320+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9845, 'log_dups_size': 0, 'ondisk_log_size': 9845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19035296, 'num_objects': 2324, 'num_object_clones': 0, 'num_object_copies': 9296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9843, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 469}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9811", 'reported_seq': 10037, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:53.132224+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:53.132224+0000', 'last_peered': '2024-09-10T20:27:53.132224+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:53.132224+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9811, 'log_dups_size': 0, 'ondisk_log_size': 9811, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18735104, 'num_objects': 2287, 'num_object_clones': 0, 'num_object_copies': 9148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9811, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 430}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9774", 'reported_seq': 9982, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:52.330588+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:52.330588+0000', 'last_peered': '2024-09-10T20:27:52.330588+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:52.330588+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9774, 'log_dups_size': 0, 'ondisk_log_size': 9774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18805920, 'num_objects': 2296, 'num_object_clones': 0, 'num_object_copies': 9184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9772, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 403}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9927", 'reported_seq': 10100, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:52.152785+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:52.152785+0000', 'last_peered': '2024-09-10T20:27:52.152785+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:52.152785+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9927, 'log_dups_size': 0, 'ondisk_log_size': 9927, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19379360, 'num_objects': 2366, 'num_object_clones': 0, 'num_object_copies': 9464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9925, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:27:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9748", 'reported_seq': 10025, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:53.461903+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:53.461903+0000', 'last_peered': '2024-09-10T20:27:53.461903+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:53.461903+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9748, 'log_dups_size': 0, 'ondisk_log_size': 9748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19369712, 'num_objects': 2365, 'num_object_clones': 0, 'num_object_copies': 9460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9745, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 509}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9837", 'reported_seq': 10138, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:53.474615+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:53.474615+0000', 'last_peered': '2024-09-10T20:27:53.474615+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:53.474615+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9837, 'log_dups_size': 0, 'ondisk_log_size': 9837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19069520, 'num_objects': 2328, 'num_object_clones': 0, 'num_object_copies': 9312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9836, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9563", 'reported_seq': 9749, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:55.151697+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:27:55.151697+0000', 'last_peered': '2024-09-10T20:27:55.151697+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:27:55.151697+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9563, 'log_dups_size': 0, 'ondisk_log_size': 9563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18578000, 'num_objects': 2268, 'num_object_clones': 0, 'num_object_copies': 9072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9562, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 365}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9786", 'reported_seq': 11066, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:53.478635+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:53.478635+0000', 'last_peered': '2024-09-10T20:27:53.478635+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:53.478635+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9786, 'log_dups_size': 0, 'ondisk_log_size': 9786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18904224, 'num_objects': 2308, 'num_object_clones': 0, 'num_object_copies': 9232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1374, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9784, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 605, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:27:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10063", 'reported_seq': 10273, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:55.137244+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:27:55.137244+0000', 'last_peered': '2024-09-10T20:27:55.137244+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:27:55.137244+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19543200, 'num_objects': 2386, 'num_object_clones': 0, 'num_object_copies': 9544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2386, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10061, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:27:57.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:27:58.408 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:58.409 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20283/148584 objects degraded (13.651%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":567,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37146,"data_bytes":304267112,"bytes_used":2885988352,"bytes_avail":383661068288,"bytes_total":386547056640,"degraded_objects":20283,"degraded_total":148584,"degraded_ratio":0.13650864157648199,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":208},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-10T20:27:40.057375+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:27:58.447 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:27:58.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:27:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:27:59.041 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:27:59.089 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-10T20:27:58.063232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":304267112,"num_objects":37146,"num_object_clones":0,"num_object_copies":148584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128072,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157323,"ondisk_log_size":157323,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818348,"kb_used_data":595788,"kb_used_omap":14,"kb_used_meta":2222449,"kb_avail":374669012,"statfs":{"total":386547056640,"available":383661068288,"internally_reserved":0,"allocated":610086912,"data_stored":609196848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2275788342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,3,7,6,22,8,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20520960,"num_objects":-2505,"num_object_clones":0,"num_object_copies":-10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004015"},"pg_stats":[{"pgid":"1.f","version":"42'9785","reported_seq":10071,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.451056+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:53.451056+0000","last_peered":"2024-09-10T20:27:53.451056+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:53.451056+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9785,"log_dups_size":0,"ondisk_log_size":9785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19144704,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9918","reported_seq":10054,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:56.258804+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:27:56.258804+0000","last_peered":"2024-09-10T20:27:56.258804+0000","last_clean":"2024-09-10T20:27:56.258804+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:27:56.258804+0000","last_undegraded":"2024-09-10T20:27:56.258804+0000","last_fullsized":"2024-09-10T20:27:56.258804+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19120128,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9814","reported_seq":10021,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.054436+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:27:55.054436+0000","last_peered":"2024-09-10T20:27:55.054436+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:27:55.054436+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19053160,"num_objects":2327,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9813,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10068","reported_seq":14242,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:56.400626+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:27:56.400626+0000","last_peered":"2024-09-10T20:27:56.400626+0000","last_clean":"2024-09-10T20:27:56.400626+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:27:56.400626+0000","last_undegraded":"2024-09-10T20:27:56.400626+0000","last_fullsized":"2024-09-10T20:27:56.400626+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19152896,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9845","reported_seq":10170,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.080320+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:53.080320+0000","last_peered":"2024-09-10T20:27:53.080320+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:53.080320+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9845,"log_dups_size":0,"ondisk_log_size":9845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19035296,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9843,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9811","reported_seq":10037,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.132224+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:53.132224+0000","last_peered":"2024-09-10T20:27:53.132224+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:53.132224+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18735104,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9848","reported_seq":10056,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:57.306471+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:57.306471+0000","last_peered":"2024-09-10T20:27:57.306471+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:57.306471+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199712,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":329}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9995","reported_seq":10165,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:57.366674+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:57.366674+0000","last_peered":"2024-09-10T20:27:57.366674+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:57.366674+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18822304,"num_objects":2298,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9670","reported_seq":13645,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:53.545273+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:53.545273+0000","last_peered":"2024-09-10T20:27:53.545273+0000","last_clean":"2024-09-10T20:27:53.545273+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:53.545273+0000","last_undegraded":"2024-09-10T20:27:53.545273+0000","last_fullsized":"2024-09-10T20:27:53.545273+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19094096,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6223,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9754","reported_seq":13728,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:55.064787+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:27:55.064787+0000","last_peered":"2024-09-10T20:27:55.064787+0000","last_clean":"2024-09-10T20:27:55.064787+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:27:55.064787+0000","last_undegraded":"2024-09-10T20:27:55.064787+0000","last_fullsized":"2024-09-10T20:27:55.064787+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19562496,"num_objects":2388,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9748","reported_seq":10025,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.461903+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:53.461903+0000","last_peered":"2024-09-10T20:27:53.461903+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:53.461903+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19369712,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9837","reported_seq":10138,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.474615+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:53.474615+0000","last_peered":"2024-09-10T20:27:53.474615+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:53.474615+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9837,"log_dups_size":0,"ondisk_log_size":9837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19069520,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9563","reported_seq":9749,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.151697+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:27:55.151697+0000","last_peered":"2024-09-10T20:27:55.151697+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:27:55.151697+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18578000,"num_objects":2268,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":365}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9786","reported_seq":11066,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:53.478635+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:53.478635+0000","last_peered":"2024-09-10T20:27:53.478635+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:53.478635+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9786,"log_dups_size":0,"ondisk_log_size":9786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904224,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9784,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10063","reported_seq":10273,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:55.137244+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:27:55.137244+0000","last_peered":"2024-09-10T20:27:55.137244+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:27:55.137244+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19543200,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10061,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9818","reported_seq":13918,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:56.503104+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:27:56.503104+0000","last_peered":"2024-09-10T20:27:56.503104+0000","last_clean":"2024-09-10T20:27:56.503104+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:27:56.503104+0000","last_undegraded":"2024-09-10T20:27:56.503104+0000","last_fullsized":"2024-09-10T20:27:56.503104+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18882560,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":691,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":304267112,"num_objects":37146,"num_object_clones":0,"num_object_copies":148584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128072,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606089216,"data_stored":606089216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157323,"ondisk_log_size":157323,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117098,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724072,"kb_used_data":150936,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93647768,"statfs":{"total":96636764160,"available":95895314432,"internally_reserved":0,"allocated":154558464,"data_stored":154335948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,6,3,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748516,"kb_used_data":149508,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93623324,"statfs":{"total":96636764160,"available":95870283776,"internally_reserved":0,"allocated":153096192,"data_stored":152873676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,10,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599156,"kb_used_data":148180,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93772684,"statfs":{"total":96636764160,"available":96023228416,"internally_reserved":0,"allocated":151736320,"data_stored":151513804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461762258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746604,"kb_used_data":147164,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93625236,"statfs":{"total":96636764160,"available":95872241664,"internally_reserved":0,"allocated":150695936,"data_stored":150473420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149946368,"data_stored":149946368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":150736896,"data_stored":150736896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151924736,"data_stored":151924736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153481216,"data_stored":153481216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:02.091 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:02.744 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:02.744 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:02.766 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-10T20:28:02.066539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":296566632,"num_objects":36206,"num_object_clones":0,"num_object_copies":144824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158063,"ondisk_log_size":158063,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2837656,"kb_used_data":583168,"kb_used_omap":14,"kb_used_meta":2254385,"kb_avail":374649704,"statfs":{"total":386547056640,"available":383641296896,"internally_reserved":0,"allocated":597164032,"data_stored":596273968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2308490806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,2,10,11,11,10,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21504000,"num_objects":-2625,"num_object_clones":0,"num_object_copies":-10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005742"},"pg_stats":[{"pgid":"1.f","version":"42'9861","reported_seq":10147,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.531973+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:58.531973+0000","last_peered":"2024-09-10T20:27:58.531973+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:58.531973+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9861,"log_dups_size":0,"ondisk_log_size":9861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18522112,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9861,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9988","reported_seq":10124,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.488580+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:01.488580+0000","last_peered":"2024-09-10T20:28:01.488580+0000","last_clean":"2024-09-10T20:28:01.488580+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:01.488580+0000","last_undegraded":"2024-09-10T20:28:01.488580+0000","last_fullsized":"2024-09-10T20:28:01.488580+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9988,"log_dups_size":0,"ondisk_log_size":9988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6429,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9889","reported_seq":10097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.174286+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:00.174286+0000","last_peered":"2024-09-10T20:28:00.174286+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:00.174286+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438760,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9888,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":344}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10128","reported_seq":14302,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.420972+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:01.420972+0000","last_peered":"2024-09-10T20:28:01.420972+0000","last_clean":"2024-09-10T20:28:01.420972+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:01.420972+0000","last_undegraded":"2024-09-10T20:28:01.420972+0000","last_fullsized":"2024-09-10T20:28:01.420972+0000","mapping_epoch":37,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9910","reported_seq":10235,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.534732+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:58.534732+0000","last_peered":"2024-09-10T20:27:58.534732+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:58.534732+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18502816,"num_objects":2259,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9876","reported_seq":10102,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.285408+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:58.285408+0000","last_peered":"2024-09-10T20:27:58.285408+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:58.285408+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18202624,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9876,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9848","reported_seq":10056,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:57.306471+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:57.306471+0000","last_peered":"2024-09-10T20:27:57.306471+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:57.306471+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199712,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":329}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9995","reported_seq":10165,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:57.366674+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:57.366674+0000","last_peered":"2024-09-10T20:27:57.366674+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:57.366674+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18822304,"num_objects":2298,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9732","reported_seq":13709,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:58.507594+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:58.507594+0000","last_peered":"2024-09-10T20:27:58.507594+0000","last_clean":"2024-09-10T20:27:58.507594+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:58.507594+0000","last_undegraded":"2024-09-10T20:27:58.507594+0000","last_fullsized":"2024-09-10T20:27:58.507594+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18586192,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6285,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9816","reported_seq":13791,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:00.144604+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:00.144604+0000","last_peered":"2024-09-10T20:28:00.144604+0000","last_clean":"2024-09-10T20:28:00.144604+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:00.144604+0000","last_undegraded":"2024-09-10T20:28:00.144604+0000","last_fullsized":"2024-09-10T20:28:00.144604+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9816,"log_dups_size":0,"ondisk_log_size":9816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19054592,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9820","reported_seq":10097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.533138+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:58.533138+0000","last_peered":"2024-09-10T20:27:58.533138+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:58.533138+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9820,"log_dups_size":0,"ondisk_log_size":9820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18779888,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9901","reported_seq":10203,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.400520+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:58.400520+0000","last_peered":"2024-09-10T20:27:58.400520+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:58.400520+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18545232,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9632","reported_seq":9820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.028814+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:00.028814+0000","last_peered":"2024-09-10T20:28:00.028814+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:00.028814+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9632,"log_dups_size":0,"ondisk_log_size":9632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18012752,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9631,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9862","reported_seq":11235,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.442600+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:58.442600+0000","last_peered":"2024-09-10T20:27:58.442600+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:58.442600+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9862,"log_dups_size":0,"ondisk_log_size":9862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18281632,"num_objects":2232,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9860,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":652,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":905}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10128","reported_seq":10338,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.079944+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:00.079944+0000","last_peered":"2024-09-10T20:28:00.079944+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:00.079944+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010720,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9877","reported_seq":13977,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.628938+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:01.628938+0000","last_peered":"2024-09-10T20:28:01.628938+0000","last_clean":"2024-09-10T20:28:01.628938+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:01.628938+0000","last_undegraded":"2024-09-10T20:28:01.628938+0000","last_fullsized":"2024-09-10T20:28:01.628938+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9877,"log_dups_size":0,"ondisk_log_size":9877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18399232,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":296566632,"num_objects":36206,"num_object_clones":0,"num_object_copies":144824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":592957440,"data_stored":592957440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158063,"ondisk_log_size":158063,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117099,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741892,"kb_used_data":146548,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93629948,"statfs":{"total":96636764160,"available":95877066752,"internally_reserved":0,"allocated":150065152,"data_stored":149842636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,7,1,6,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731824,"kb_used_data":145304,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93640016,"statfs":{"total":96636764160,"available":95887376384,"internally_reserved":0,"allocated":148791296,"data_stored":148568780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617336,"kb_used_data":144152,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93754504,"statfs":{"total":96636764160,"available":96004612096,"internally_reserved":0,"allocated":147611648,"data_stored":147389132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484503250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,0,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746604,"kb_used_data":147164,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93625236,"statfs":{"total":96636764160,"available":95872241664,"internally_reserved":0,"allocated":150695936,"data_stored":150473420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149946368,"data_stored":149946368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146423808,"data_stored":146423808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147574784,"data_stored":147574784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149012480,"data_stored":149012480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:02.767 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:03.407 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:03.408 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:03.431 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-10T20:28:02.066539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":296566632,"num_objects":36206,"num_object_clones":0,"num_object_copies":144824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158063,"ondisk_log_size":158063,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2837656,"kb_used_data":583168,"kb_used_omap":14,"kb_used_meta":2254385,"kb_avail":374649704,"statfs":{"total":386547056640,"available":383641296896,"internally_reserved":0,"allocated":597164032,"data_stored":596273968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2308490806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,2,10,11,11,10,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21504000,"num_objects":-2625,"num_object_clones":0,"num_object_copies":-10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005742"},"pg_stats":[{"pgid":"1.f","version":"42'9861","reported_seq":10147,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.531973+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:27:58.531973+0000","last_peered":"2024-09-10T20:27:58.531973+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:27:58.531973+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9861,"log_dups_size":0,"ondisk_log_size":9861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18522112,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9861,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9988","reported_seq":10124,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.488580+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:01.488580+0000","last_peered":"2024-09-10T20:28:01.488580+0000","last_clean":"2024-09-10T20:28:01.488580+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:01.488580+0000","last_undegraded":"2024-09-10T20:28:01.488580+0000","last_fullsized":"2024-09-10T20:28:01.488580+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9988,"log_dups_size":0,"ondisk_log_size":9988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6429,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9889","reported_seq":10097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.174286+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:00.174286+0000","last_peered":"2024-09-10T20:28:00.174286+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:00.174286+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438760,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9888,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":344}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10128","reported_seq":14302,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.420972+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:01.420972+0000","last_peered":"2024-09-10T20:28:01.420972+0000","last_clean":"2024-09-10T20:28:01.420972+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:01.420972+0000","last_undegraded":"2024-09-10T20:28:01.420972+0000","last_fullsized":"2024-09-10T20:28:01.420972+0000","mapping_epoch":37,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9910","reported_seq":10235,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.534732+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:27:58.534732+0000","last_peered":"2024-09-10T20:27:58.534732+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:27:58.534732+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18502816,"num_objects":2259,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9876","reported_seq":10102,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.285408+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:27:58.285408+0000","last_peered":"2024-09-10T20:27:58.285408+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:27:58.285408+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18202624,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9876,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9848","reported_seq":10056,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:57.306471+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:27:57.306471+0000","last_peered":"2024-09-10T20:27:57.306471+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:27:57.306471+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199712,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":329}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'9995","reported_seq":10165,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:57.366674+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:27:57.366674+0000","last_peered":"2024-09-10T20:27:57.366674+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:27:57.366674+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18822304,"num_objects":2298,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9732","reported_seq":13709,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:27:58.507594+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:27:58.507594+0000","last_peered":"2024-09-10T20:27:58.507594+0000","last_clean":"2024-09-10T20:27:58.507594+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:27:58.507594+0000","last_undegraded":"2024-09-10T20:27:58.507594+0000","last_fullsized":"2024-09-10T20:27:58.507594+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18586192,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6285,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9816","reported_seq":13791,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:00.144604+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:00.144604+0000","last_peered":"2024-09-10T20:28:00.144604+0000","last_clean":"2024-09-10T20:28:00.144604+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:00.144604+0000","last_undegraded":"2024-09-10T20:28:00.144604+0000","last_fullsized":"2024-09-10T20:28:00.144604+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9816,"log_dups_size":0,"ondisk_log_size":9816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19054592,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9820","reported_seq":10097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.533138+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:27:58.533138+0000","last_peered":"2024-09-10T20:27:58.533138+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:27:58.533138+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9820,"log_dups_size":0,"ondisk_log_size":9820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18779888,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9901","reported_seq":10203,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.400520+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:27:58.400520+0000","last_peered":"2024-09-10T20:27:58.400520+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:27:58.400520+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18545232,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9632","reported_seq":9820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.028814+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:00.028814+0000","last_peered":"2024-09-10T20:28:00.028814+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:00.028814+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9632,"log_dups_size":0,"ondisk_log_size":9632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18012752,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9631,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9862","reported_seq":11235,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:27:58.442600+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:27:58.442600+0000","last_peered":"2024-09-10T20:27:58.442600+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:27:58.442600+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9862,"log_dups_size":0,"ondisk_log_size":9862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18281632,"num_objects":2232,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9860,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":652,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":905}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10128","reported_seq":10338,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.079944+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:00.079944+0000","last_peered":"2024-09-10T20:28:00.079944+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:00.079944+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010720,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9877","reported_seq":13977,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.628938+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:01.628938+0000","last_peered":"2024-09-10T20:28:01.628938+0000","last_clean":"2024-09-10T20:28:01.628938+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:01.628938+0000","last_undegraded":"2024-09-10T20:28:01.628938+0000","last_fullsized":"2024-09-10T20:28:01.628938+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9877,"log_dups_size":0,"ondisk_log_size":9877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18399232,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":296566632,"num_objects":36206,"num_object_clones":0,"num_object_copies":144824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":592957440,"data_stored":592957440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158063,"ondisk_log_size":158063,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117099,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741892,"kb_used_data":146548,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93629948,"statfs":{"total":96636764160,"available":95877066752,"internally_reserved":0,"allocated":150065152,"data_stored":149842636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,7,1,6,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731824,"kb_used_data":145304,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93640016,"statfs":{"total":96636764160,"available":95887376384,"internally_reserved":0,"allocated":148791296,"data_stored":148568780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617336,"kb_used_data":144152,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93754504,"statfs":{"total":96636764160,"available":96004612096,"internally_reserved":0,"allocated":147611648,"data_stored":147389132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484503250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,0,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746604,"kb_used_data":147164,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93625236,"statfs":{"total":96636764160,"available":95872241664,"internally_reserved":0,"allocated":150695936,"data_stored":150473420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149946368,"data_stored":149946368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146423808,"data_stored":146423808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147574784,"data_stored":147574784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149012480,"data_stored":149012480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9861", 'reported_seq': 10147, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:58.531973+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:27:58.531973+0000', 'last_peered': '2024-09-10T20:27:58.531973+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:27:58.531973+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9861, 'log_dups_size': 0, 'ondisk_log_size': 9861, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18522112, 'num_objects': 2261, 'num_object_clones': 0, 'num_object_copies': 9044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9861, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 406}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9889", 'reported_seq': 10097, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:00.174286+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:00.174286+0000', 'last_peered': '2024-09-10T20:28:00.174286+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:00.174286+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9889, 'log_dups_size': 0, 'ondisk_log_size': 9889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18438760, 'num_objects': 2252, 'num_object_clones': 0, 'num_object_copies': 9008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2252, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9888, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 344}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9910", 'reported_seq': 10235, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:58.534732+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:27:58.534732+0000', 'last_peered': '2024-09-10T20:27:58.534732+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:27:58.534732+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9910, 'log_dups_size': 0, 'ondisk_log_size': 9910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18502816, 'num_objects': 2259, 'num_object_clones': 0, 'num_object_copies': 9036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9908, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 404}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9876", 'reported_seq': 10102, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:58.285408+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:27:58.285408+0000', 'last_peered': '2024-09-10T20:27:58.285408+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:27:58.285408+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9876, 'log_dups_size': 0, 'ondisk_log_size': 9876, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18202624, 'num_objects': 2222, 'num_object_clones': 0, 'num_object_copies': 8888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9876, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 365}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:03.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9848", 'reported_seq': 10056, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:57.306471+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:27:57.306471+0000', 'last_peered': '2024-09-10T20:27:57.306471+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:27:57.306471+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9848, 'log_dups_size': 0, 'ondisk_log_size': 9848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18199712, 'num_objects': 2222, 'num_object_clones': 0, 'num_object_copies': 8888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9846, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 329}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'9995", 'reported_seq': 10165, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:57.366674+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:27:57.366674+0000', 'last_peered': '2024-09-10T20:27:57.366674+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:27:57.366674+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9995, 'log_dups_size': 0, 'ondisk_log_size': 9995, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18822304, 'num_objects': 2298, 'num_object_clones': 0, 'num_object_copies': 9192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9993, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 298}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9820", 'reported_seq': 10097, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:58.533138+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:27:58.533138+0000', 'last_peered': '2024-09-10T20:27:58.533138+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:27:58.533138+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9820, 'log_dups_size': 0, 'ondisk_log_size': 9820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18779888, 'num_objects': 2293, 'num_object_clones': 0, 'num_object_copies': 9172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9817, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 437}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9901", 'reported_seq': 10203, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:58.400520+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:27:58.400520+0000', 'last_peered': '2024-09-10T20:27:58.400520+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:27:58.400520+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9901, 'log_dups_size': 0, 'ondisk_log_size': 9901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18545232, 'num_objects': 2264, 'num_object_clones': 0, 'num_object_copies': 9056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9900, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9632", 'reported_seq': 9820, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:00.028814+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:00.028814+0000', 'last_peered': '2024-09-10T20:28:00.028814+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:00.028814+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9632, 'log_dups_size': 0, 'ondisk_log_size': 9632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18012752, 'num_objects': 2199, 'num_object_clones': 0, 'num_object_copies': 8796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9631, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:03.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9862", 'reported_seq': 11235, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:27:58.442600+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:27:58.442600+0000', 'last_peered': '2024-09-10T20:27:58.442600+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:27:58.442600+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9862, 'log_dups_size': 0, 'ondisk_log_size': 9862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18281632, 'num_objects': 2232, 'num_object_clones': 0, 'num_object_copies': 8928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9860, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 652, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 905}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:03.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:03.435 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10128", 'reported_seq': 10338, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:00.079944+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:00.079944+0000', 'last_peered': '2024-09-10T20:28:00.079944+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:00.079944+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19010720, 'num_objects': 2321, 'num_object_clones': 0, 'num_object_copies': 9284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10126, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:03.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:04.185 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:04.185 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20236/144824 objects degraded (13.973%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":572,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36206,"data_bytes":296566632,"bytes_used":2905759744,"bytes_avail":383641296896,"bytes_total":386547056640,"degraded_objects":20236,"degraded_total":144824,"degraded_ratio":0.13972822184168371,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-10T20:27:40.057375+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:28:04.211 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:04.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:04.774 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:04.774 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:04.796 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-10T20:28:04.067181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":291626856,"num_objects":35603,"num_object_clones":0,"num_object_copies":142412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129615,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158666,"ondisk_log_size":158666,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2820932,"kb_used_data":574760,"kb_used_omap":14,"kb_used_meta":2246065,"kb_avail":374666428,"statfs":{"total":386547056640,"available":383658422272,"internally_reserved":0,"allocated":588554240,"data_stored":587664176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2299971126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,13,9,18,12,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22454272,"num_objects":-2741,"num_object_clones":0,"num_object_copies":-10964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005710"},"pg_stats":[{"pgid":"1.f","version":"42'9933","reported_seq":10219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.551950+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:03.551950+0000","last_peered":"2024-09-10T20:28:03.551950+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:03.551950+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9933,"log_dups_size":0,"ondisk_log_size":9933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17932288,"num_objects":2189,"num_object_clones":0,"num_object_copies":8756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'9988","reported_seq":10124,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.488580+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:01.488580+0000","last_peered":"2024-09-10T20:28:01.488580+0000","last_clean":"2024-09-10T20:28:01.488580+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:01.488580+0000","last_undegraded":"2024-09-10T20:28:01.488580+0000","last_fullsized":"2024-09-10T20:28:01.488580+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":9988,"log_dups_size":0,"ondisk_log_size":9988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6429,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9889","reported_seq":10097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.174286+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:00.174286+0000","last_peered":"2024-09-10T20:28:00.174286+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:00.174286+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438760,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9888,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":344}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10128","reported_seq":14302,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.420972+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:01.420972+0000","last_peered":"2024-09-10T20:28:01.420972+0000","last_clean":"2024-09-10T20:28:01.420972+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:01.420972+0000","last_undegraded":"2024-09-10T20:28:01.420972+0000","last_fullsized":"2024-09-10T20:28:01.420972+0000","mapping_epoch":37,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9977","reported_seq":10302,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.328685+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:03.328685+0000","last_peered":"2024-09-10T20:28:03.328685+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:03.328685+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17953952,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9975,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9945","reported_seq":10171,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.240960+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:03.240960+0000","last_peered":"2024-09-10T20:28:03.240960+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:03.240960+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17637376,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9945,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9914","reported_seq":10121,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:02.424801+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:02.424801+0000","last_peered":"2024-09-10T20:28:02.424801+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:02.424801+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9914,"log_dups_size":0,"ondisk_log_size":9914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17659040,"num_objects":2156,"num_object_clones":0,"num_object_copies":8624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9912,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":263}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10052","reported_seq":10225,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:02.065362+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:02.065362+0000","last_peered":"2024-09-10T20:28:02.065362+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:02.065362+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355360,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9797","reported_seq":13774,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:03.522880+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:03.522880+0000","last_peered":"2024-09-10T20:28:03.522880+0000","last_clean":"2024-09-10T20:28:03.522880+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:03.522880+0000","last_undegraded":"2024-09-10T20:28:03.522880+0000","last_fullsized":"2024-09-10T20:28:03.522880+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9797,"log_dups_size":0,"ondisk_log_size":9797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18053712,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9816","reported_seq":13791,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:00.144604+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:00.144604+0000","last_peered":"2024-09-10T20:28:00.144604+0000","last_clean":"2024-09-10T20:28:00.144604+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:00.144604+0000","last_undegraded":"2024-09-10T20:28:00.144604+0000","last_fullsized":"2024-09-10T20:28:00.144604+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9816,"log_dups_size":0,"ondisk_log_size":9816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19054592,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9894","reported_seq":10172,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.518732+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:03.518732+0000","last_peered":"2024-09-10T20:28:03.518732+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:03.518732+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18173680,"num_objects":2219,"num_object_clones":0,"num_object_copies":8876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9965","reported_seq":10267,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.602186+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:03.602186+0000","last_peered":"2024-09-10T20:28:03.602186+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:03.602186+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18020944,"num_objects":2200,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9964,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9632","reported_seq":9820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.028814+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:00.028814+0000","last_peered":"2024-09-10T20:28:00.028814+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:00.028814+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9632,"log_dups_size":0,"ondisk_log_size":9632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18012752,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9631,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9931","reported_seq":11455,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.542369+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:03.542369+0000","last_peered":"2024-09-10T20:28:03.542369+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:03.542369+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17716384,"num_objects":2163,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9929,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":912}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10128","reported_seq":10338,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:00.079944+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:00.079944+0000","last_peered":"2024-09-10T20:28:00.079944+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:00.079944+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010720,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":413}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9877","reported_seq":13977,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:01.628938+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:01.628938+0000","last_peered":"2024-09-10T20:28:01.628938+0000","last_clean":"2024-09-10T20:28:01.628938+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:01.628938+0000","last_undegraded":"2024-09-10T20:28:01.628938+0000","last_fullsized":"2024-09-10T20:28:01.628938+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9877,"log_dups_size":0,"ondisk_log_size":9877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18399232,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":291626856,"num_objects":35603,"num_object_clones":0,"num_object_copies":142412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129615,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":584241152,"data_stored":584241152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158666,"ondisk_log_size":158666,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741840,"kb_used_data":142332,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93630000,"statfs":{"total":96636764160,"available":95877120000,"internally_reserved":0,"allocated":145747968,"data_stored":145525452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,8,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731824,"kb_used_data":145304,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93640016,"statfs":{"total":96636764160,"available":95887376384,"internally_reserved":0,"allocated":148791296,"data_stored":148568780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617336,"kb_used_data":144152,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93754504,"statfs":{"total":96636764160,"available":96004612096,"internally_reserved":0,"allocated":147611648,"data_stored":147389132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484503250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,0,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729932,"kb_used_data":142972,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93641908,"statfs":{"total":96636764160,"available":95889313792,"internally_reserved":0,"allocated":146403328,"data_stored":146180812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145600512,"data_stored":145600512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146423808,"data_stored":146423808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147574784,"data_stored":147574784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144642048,"data_stored":144642048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:07.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:08.429 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:08.462 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-10T20:28:08.068881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286769000,"num_objects":35010,"num_object_clones":0,"num_object_copies":140040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130208,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159059,"ondisk_log_size":159059,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803244,"kb_used_data":561240,"kb_used_omap":14,"kb_used_meta":2241905,"kb_avail":374684116,"statfs":{"total":386547056640,"available":383676534784,"internally_reserved":0,"allocated":574709760,"data_stored":573819696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2295711286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,6,11,18,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20250624,"num_objects":-2472,"num_object_clones":0,"num_object_copies":-9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006108"},"pg_stats":[{"pgid":"1.f","version":"42'9933","reported_seq":10219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.551950+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:03.551950+0000","last_peered":"2024-09-10T20:28:03.551950+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:03.551950+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9933,"log_dups_size":0,"ondisk_log_size":9933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17932288,"num_objects":2189,"num_object_clones":0,"num_object_copies":8756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10046","reported_seq":10182,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.652824+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:06.652824+0000","last_peered":"2024-09-10T20:28:06.652824+0000","last_clean":"2024-09-10T20:28:06.652824+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:06.652824+0000","last_undegraded":"2024-09-10T20:28:06.652824+0000","last_fullsized":"2024-09-10T20:28:06.652824+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18071552,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9961","reported_seq":10169,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.062325+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:05.062325+0000","last_peered":"2024-09-10T20:28:05.062325+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:05.062325+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17848936,"num_objects":2180,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9960,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":272}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10216","reported_seq":14390,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.625021+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:06.625021+0000","last_peered":"2024-09-10T20:28:06.625021+0000","last_clean":"2024-09-10T20:28:06.625021+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:06.625021+0000","last_undegraded":"2024-09-10T20:28:06.625021+0000","last_fullsized":"2024-09-10T20:28:06.625021+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17940480,"num_objects":2190,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9977","reported_seq":10302,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.328685+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:03.328685+0000","last_peered":"2024-09-10T20:28:03.328685+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:03.328685+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17953952,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9975,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9945","reported_seq":10171,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.240960+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:03.240960+0000","last_peered":"2024-09-10T20:28:03.240960+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:03.240960+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17637376,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9945,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9974","reported_seq":10182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:07.512515+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:07.512515+0000","last_peered":"2024-09-10T20:28:07.512515+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:07.512515+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9974,"log_dups_size":0,"ondisk_log_size":9974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9972,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10126","reported_seq":10298,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:07.188701+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:07.188701+0000","last_peered":"2024-09-10T20:28:07.188701+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:07.188701+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17749152,"num_objects":2167,"num_object_clones":0,"num_object_copies":8668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10124,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9797","reported_seq":13774,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:03.522880+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:03.522880+0000","last_peered":"2024-09-10T20:28:03.522880+0000","last_clean":"2024-09-10T20:28:03.522880+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:03.522880+0000","last_undegraded":"2024-09-10T20:28:03.522880+0000","last_fullsized":"2024-09-10T20:28:03.522880+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9797,"log_dups_size":0,"ondisk_log_size":9797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18053712,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9880","reported_seq":13855,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:05.127523+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:05.127523+0000","last_peered":"2024-09-10T20:28:05.127523+0000","last_clean":"2024-09-10T20:28:05.127523+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:05.127523+0000","last_undegraded":"2024-09-10T20:28:05.127523+0000","last_fullsized":"2024-09-10T20:28:05.127523+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18530304,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2518,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9894","reported_seq":10172,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.518732+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:03.518732+0000","last_peered":"2024-09-10T20:28:03.518732+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:03.518732+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18173680,"num_objects":2219,"num_object_clones":0,"num_object_copies":8876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9965","reported_seq":10267,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.602186+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:03.602186+0000","last_peered":"2024-09-10T20:28:03.602186+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:03.602186+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18020944,"num_objects":2200,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9964,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9690","reported_seq":9878,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.060998+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:05.060998+0000","last_peered":"2024-09-10T20:28:05.060998+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:05.060998+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17537616,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9931","reported_seq":11455,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.542369+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:03.542369+0000","last_peered":"2024-09-10T20:28:03.542369+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:03.542369+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17716384,"num_objects":2163,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9929,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":912}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10187","reported_seq":10397,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.138229+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:05.138229+0000","last_peered":"2024-09-10T20:28:05.138229+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:05.138229+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527392,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9937","reported_seq":14036,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.738803+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:06.738803+0000","last_peered":"2024-09-10T20:28:06.738803+0000","last_clean":"2024-09-10T20:28:06.738803+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:06.738803+0000","last_undegraded":"2024-09-10T20:28:06.738803+0000","last_fullsized":"2024-09-10T20:28:06.738803+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17907712,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":286769000,"num_objects":35010,"num_object_clones":0,"num_object_copies":140040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130208,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":571342848,"data_stored":571342848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159059,"ondisk_log_size":159059,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741840,"kb_used_data":142332,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93630000,"statfs":{"total":96636764160,"available":95877120000,"internally_reserved":0,"allocated":145747968,"data_stored":145525452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,8,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731772,"kb_used_data":141096,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93640068,"statfs":{"total":96636764160,"available":95887429632,"internally_reserved":0,"allocated":144482304,"data_stored":144259788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599784,"kb_used_data":139092,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93772056,"statfs":{"total":96636764160,"available":96022585344,"internally_reserved":0,"allocated":142430208,"data_stored":142207692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729848,"kb_used_data":138720,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93641992,"statfs":{"total":96636764160,"available":95889399808,"internally_reserved":0,"allocated":142049280,"data_stored":141826764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141295616,"data_stored":141295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142041088,"data_stored":142041088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143364096,"data_stored":143364096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144642048,"data_stored":144642048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:08.463 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:09.064 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:09.096 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-10T20:28:08.068881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286769000,"num_objects":35010,"num_object_clones":0,"num_object_copies":140040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130208,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159059,"ondisk_log_size":159059,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803244,"kb_used_data":561240,"kb_used_omap":14,"kb_used_meta":2241905,"kb_avail":374684116,"statfs":{"total":386547056640,"available":383676534784,"internally_reserved":0,"allocated":574709760,"data_stored":573819696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2295711286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,6,11,18,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20250624,"num_objects":-2472,"num_object_clones":0,"num_object_copies":-9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006108"},"pg_stats":[{"pgid":"1.f","version":"42'9933","reported_seq":10219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.551950+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:03.551950+0000","last_peered":"2024-09-10T20:28:03.551950+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:03.551950+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9933,"log_dups_size":0,"ondisk_log_size":9933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17932288,"num_objects":2189,"num_object_clones":0,"num_object_copies":8756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10046","reported_seq":10182,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.652824+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:06.652824+0000","last_peered":"2024-09-10T20:28:06.652824+0000","last_clean":"2024-09-10T20:28:06.652824+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:06.652824+0000","last_undegraded":"2024-09-10T20:28:06.652824+0000","last_fullsized":"2024-09-10T20:28:06.652824+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18071552,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9961","reported_seq":10169,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.062325+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:05.062325+0000","last_peered":"2024-09-10T20:28:05.062325+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:05.062325+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17848936,"num_objects":2180,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9960,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":272}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10216","reported_seq":14390,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.625021+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:06.625021+0000","last_peered":"2024-09-10T20:28:06.625021+0000","last_clean":"2024-09-10T20:28:06.625021+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:06.625021+0000","last_undegraded":"2024-09-10T20:28:06.625021+0000","last_fullsized":"2024-09-10T20:28:06.625021+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17940480,"num_objects":2190,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'9977","reported_seq":10302,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.328685+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:03.328685+0000","last_peered":"2024-09-10T20:28:03.328685+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:03.328685+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17953952,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9975,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'9945","reported_seq":10171,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.240960+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:03.240960+0000","last_peered":"2024-09-10T20:28:03.240960+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:03.240960+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17637376,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9945,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9974","reported_seq":10182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:07.512515+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:07.512515+0000","last_peered":"2024-09-10T20:28:07.512515+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:07.512515+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9974,"log_dups_size":0,"ondisk_log_size":9974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9972,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10126","reported_seq":10298,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:07.188701+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:07.188701+0000","last_peered":"2024-09-10T20:28:07.188701+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:07.188701+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17749152,"num_objects":2167,"num_object_clones":0,"num_object_copies":8668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10124,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9797","reported_seq":13774,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:03.522880+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:03.522880+0000","last_peered":"2024-09-10T20:28:03.522880+0000","last_clean":"2024-09-10T20:28:03.522880+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:03.522880+0000","last_undegraded":"2024-09-10T20:28:03.522880+0000","last_fullsized":"2024-09-10T20:28:03.522880+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9797,"log_dups_size":0,"ondisk_log_size":9797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18053712,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9880","reported_seq":13855,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:05.127523+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:05.127523+0000","last_peered":"2024-09-10T20:28:05.127523+0000","last_clean":"2024-09-10T20:28:05.127523+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:05.127523+0000","last_undegraded":"2024-09-10T20:28:05.127523+0000","last_fullsized":"2024-09-10T20:28:05.127523+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18530304,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2518,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9894","reported_seq":10172,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.518732+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:03.518732+0000","last_peered":"2024-09-10T20:28:03.518732+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:03.518732+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18173680,"num_objects":2219,"num_object_clones":0,"num_object_copies":8876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'9965","reported_seq":10267,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.602186+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:03.602186+0000","last_peered":"2024-09-10T20:28:03.602186+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:03.602186+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18020944,"num_objects":2200,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9964,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9690","reported_seq":9878,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.060998+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:05.060998+0000","last_peered":"2024-09-10T20:28:05.060998+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:05.060998+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17537616,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9931","reported_seq":11455,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:03.542369+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:03.542369+0000","last_peered":"2024-09-10T20:28:03.542369+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:03.542369+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17716384,"num_objects":2163,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9929,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":912}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10187","reported_seq":10397,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.138229+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:05.138229+0000","last_peered":"2024-09-10T20:28:05.138229+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:05.138229+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527392,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9937","reported_seq":14036,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.738803+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:06.738803+0000","last_peered":"2024-09-10T20:28:06.738803+0000","last_clean":"2024-09-10T20:28:06.738803+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:06.738803+0000","last_undegraded":"2024-09-10T20:28:06.738803+0000","last_fullsized":"2024-09-10T20:28:06.738803+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17907712,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":286769000,"num_objects":35010,"num_object_clones":0,"num_object_copies":140040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130208,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":571342848,"data_stored":571342848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159059,"ondisk_log_size":159059,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741840,"kb_used_data":142332,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93630000,"statfs":{"total":96636764160,"available":95877120000,"internally_reserved":0,"allocated":145747968,"data_stored":145525452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,8,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731772,"kb_used_data":141096,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93640068,"statfs":{"total":96636764160,"available":95887429632,"internally_reserved":0,"allocated":144482304,"data_stored":144259788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599784,"kb_used_data":139092,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93772056,"statfs":{"total":96636764160,"available":96022585344,"internally_reserved":0,"allocated":142430208,"data_stored":142207692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729848,"kb_used_data":138720,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93641992,"statfs":{"total":96636764160,"available":95889399808,"internally_reserved":0,"allocated":142049280,"data_stored":141826764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141295616,"data_stored":141295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142041088,"data_stored":142041088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143364096,"data_stored":143364096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144642048,"data_stored":144642048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'9933", 'reported_seq': 10219, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:03.551950+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:03.551950+0000', 'last_peered': '2024-09-10T20:28:03.551950+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:03.551950+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9933, 'log_dups_size': 0, 'ondisk_log_size': 9933, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17932288, 'num_objects': 2189, 'num_object_clones': 0, 'num_object_copies': 8756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9933, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 334}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'9961", 'reported_seq': 10169, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:05.062325+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:05.062325+0000', 'last_peered': '2024-09-10T20:28:05.062325+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:05.062325+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9961, 'log_dups_size': 0, 'ondisk_log_size': 9961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17848936, 'num_objects': 2180, 'num_object_clones': 0, 'num_object_copies': 8720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2180, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9960, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 272}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'9977", 'reported_seq': 10302, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:03.328685+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:03.328685+0000', 'last_peered': '2024-09-10T20:28:03.328685+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:03.328685+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9977, 'log_dups_size': 0, 'ondisk_log_size': 9977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17953952, 'num_objects': 2192, 'num_object_clones': 0, 'num_object_copies': 8768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9975, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 337}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:09.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'9945", 'reported_seq': 10171, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:03.240960+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:03.240960+0000', 'last_peered': '2024-09-10T20:28:03.240960+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:03.240960+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9945, 'log_dups_size': 0, 'ondisk_log_size': 9945, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17637376, 'num_objects': 2153, 'num_object_clones': 0, 'num_object_copies': 8612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9945, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 296}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'9974", 'reported_seq': 10182, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:07.512515+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:07.512515+0000', 'last_peered': '2024-09-10T20:28:07.512515+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:07.512515+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9974, 'log_dups_size': 0, 'ondisk_log_size': 9974, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17167520, 'num_objects': 2096, 'num_object_clones': 0, 'num_object_copies': 8384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9972, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 203}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10126", 'reported_seq': 10298, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:07.188701+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:07.188701+0000', 'last_peered': '2024-09-10T20:28:07.188701+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:07.188701+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17749152, 'num_objects': 2167, 'num_object_clones': 0, 'num_object_copies': 8668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10124, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'9894", 'reported_seq': 10172, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:03.518732+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:03.518732+0000', 'last_peered': '2024-09-10T20:28:03.518732+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:03.518732+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9894, 'log_dups_size': 0, 'ondisk_log_size': 9894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18173680, 'num_objects': 2219, 'num_object_clones': 0, 'num_object_copies': 8876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9891, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 363}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'9965", 'reported_seq': 10267, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:03.602186+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:03.602186+0000', 'last_peered': '2024-09-10T20:28:03.602186+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:03.602186+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9965, 'log_dups_size': 0, 'ondisk_log_size': 9965, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18020944, 'num_objects': 2200, 'num_object_clones': 0, 'num_object_copies': 8800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9964, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9690", 'reported_seq': 9878, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:05.060998+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:05.060998+0000', 'last_peered': '2024-09-10T20:28:05.060998+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:05.060998+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9690, 'log_dups_size': 0, 'ondisk_log_size': 9690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17537616, 'num_objects': 2141, 'num_object_clones': 0, 'num_object_copies': 8564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9689, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:09.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:09.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'9931", 'reported_seq': 11455, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:03.542369+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:03.542369+0000', 'last_peered': '2024-09-10T20:28:03.542369+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:03.542369+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9931, 'log_dups_size': 0, 'ondisk_log_size': 9931, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17716384, 'num_objects': 2163, 'num_object_clones': 0, 'num_object_copies': 8652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9929, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 728, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 912}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:09.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:09.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10187", 'reported_seq': 10397, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:05.138229+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:05.138229+0000', 'last_peered': '2024-09-10T20:28:05.138229+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:05.138229+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18527392, 'num_objects': 2262, 'num_object_clones': 0, 'num_object_copies': 9048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10185, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 354}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:09.099 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:09.839 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:09.839 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20160/140040 objects degraded (14.396%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":578,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35010,"data_bytes":286769000,"bytes_used":2870521856,"bytes_avail":383676534784,"bytes_total":386547056640,"degraded_objects":20160,"degraded_total":140040,"degraded_ratio":0.14395886889460155,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":205},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:09.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:10.474 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:10.475 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:10.502 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-10T20:28:10.069684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":282959720,"num_objects":34545,"num_object_clones":0,"num_object_copies":138180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130673,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1591,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159524,"ondisk_log_size":159524,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786492,"kb_used_data":556960,"kb_used_omap":14,"kb_used_meta":2229425,"kb_avail":374700868,"statfs":{"total":386547056640,"available":383693688832,"internally_reserved":0,"allocated":570327040,"data_stored":569436976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2282931766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,6,10,14,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21307392,"num_objects":-2601,"num_object_clones":0,"num_object_copies":-10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2601,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006451"},"pg_stats":[{"pgid":"1.f","version":"42'9999","reported_seq":10285,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:08.656874+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:08.656874+0000","last_peered":"2024-09-10T20:28:08.656874+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:08.656874+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17391616,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10046","reported_seq":10182,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.652824+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:06.652824+0000","last_peered":"2024-09-10T20:28:06.652824+0000","last_clean":"2024-09-10T20:28:06.652824+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:06.652824+0000","last_undegraded":"2024-09-10T20:28:06.652824+0000","last_fullsized":"2024-09-10T20:28:06.652824+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18071552,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'9961","reported_seq":10169,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.062325+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:05.062325+0000","last_peered":"2024-09-10T20:28:05.062325+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:05.062325+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17848936,"num_objects":2180,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9960,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":272}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10216","reported_seq":14390,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.625021+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:06.625021+0000","last_peered":"2024-09-10T20:28:06.625021+0000","last_clean":"2024-09-10T20:28:06.625021+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:06.625021+0000","last_undegraded":"2024-09-10T20:28:06.625021+0000","last_fullsized":"2024-09-10T20:28:06.625021+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17940480,"num_objects":2190,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10056","reported_seq":10380,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:08.691069+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:08.691069+0000","last_peered":"2024-09-10T20:28:08.691069+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:08.691069+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17306784,"num_objects":2113,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10009","reported_seq":10234,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:08.622307+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:08.622307+0000","last_peered":"2024-09-10T20:28:08.622307+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:08.622307+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17113088,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10009,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'9974","reported_seq":10182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:07.512515+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:07.512515+0000","last_peered":"2024-09-10T20:28:07.512515+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:07.512515+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9974,"log_dups_size":0,"ondisk_log_size":9974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9972,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10126","reported_seq":10298,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:07.188701+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:07.188701+0000","last_peered":"2024-09-10T20:28:07.188701+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:07.188701+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17749152,"num_objects":2167,"num_object_clones":0,"num_object_copies":8668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10124,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9869","reported_seq":13846,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:08.604726+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:08.604726+0000","last_peered":"2024-09-10T20:28:08.604726+0000","last_clean":"2024-09-10T20:28:08.604726+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:08.604726+0000","last_undegraded":"2024-09-10T20:28:08.604726+0000","last_fullsized":"2024-09-10T20:28:08.604726+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9869,"log_dups_size":0,"ondisk_log_size":9869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17463888,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6422,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9880","reported_seq":13855,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:05.127523+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:05.127523+0000","last_peered":"2024-09-10T20:28:05.127523+0000","last_clean":"2024-09-10T20:28:05.127523+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:05.127523+0000","last_undegraded":"2024-09-10T20:28:05.127523+0000","last_fullsized":"2024-09-10T20:28:05.127523+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18530304,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2518,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'9954","reported_seq":10232,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:08.606699+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:08.606699+0000","last_peered":"2024-09-10T20:28:08.606699+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:08.606699+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9954,"log_dups_size":0,"ondisk_log_size":9954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17682160,"num_objects":2159,"num_object_clones":0,"num_object_copies":8636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9951,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10024","reported_seq":10325,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:08.611881+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:08.611881+0000","last_peered":"2024-09-10T20:28:08.611881+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:08.611881+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17537616,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9690","reported_seq":9878,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.060998+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:05.060998+0000","last_peered":"2024-09-10T20:28:05.060998+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:05.060998+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17537616,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'9996","reported_seq":11638,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:08.746697+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:08.746697+0000","last_peered":"2024-09-10T20:28:08.746697+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:08.746697+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9996,"log_dups_size":0,"ondisk_log_size":9996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17183904,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9994,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":905}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10187","reported_seq":10397,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:05.138229+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:05.138229+0000","last_peered":"2024-09-10T20:28:05.138229+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:05.138229+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527392,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9937","reported_seq":14036,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:06.738803+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:06.738803+0000","last_peered":"2024-09-10T20:28:06.738803+0000","last_clean":"2024-09-10T20:28:06.738803+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:06.738803+0000","last_undegraded":"2024-09-10T20:28:06.738803+0000","last_fullsized":"2024-09-10T20:28:06.738803+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17907712,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":282959720,"num_objects":34545,"num_object_clones":0,"num_object_copies":138180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130673,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1591,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":566960128,"data_stored":566960128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159524,"ondisk_log_size":159524,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117101,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725088,"kb_used_data":138052,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93646752,"statfs":{"total":96636764160,"available":95894274048,"internally_reserved":0,"allocated":141365248,"data_stored":141142732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":601091792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731772,"kb_used_data":141096,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93640068,"statfs":{"total":96636764160,"available":95887429632,"internally_reserved":0,"allocated":144482304,"data_stored":144259788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599784,"kb_used_data":139092,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93772056,"statfs":{"total":96636764160,"available":96022585344,"internally_reserved":0,"allocated":142430208,"data_stored":142207692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729848,"kb_used_data":138720,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93641992,"statfs":{"total":96636764160,"available":95889399808,"internally_reserved":0,"allocated":142049280,"data_stored":141826764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141295616,"data_stored":141295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142041088,"data_stored":142041088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143364096,"data_stored":143364096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140259328,"data_stored":140259328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:13.504 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:14.098 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:14.099 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:14.110 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-10T20:28:14.071213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274333544,"num_objects":33492,"num_object_clones":0,"num_object_copies":133968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131726,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160277,"ondisk_log_size":160277,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786308,"kb_used_data":540144,"kb_used_omap":14,"kb_used_meta":2246065,"kb_avail":374701052,"statfs":{"total":386547056640,"available":383693877248,"internally_reserved":0,"allocated":553107456,"data_stored":552217392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2299971126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,4,21,25,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22233088,"num_objects":-2714,"num_object_clones":0,"num_object_copies":-10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004674"},"pg_stats":[{"pgid":"1.f","version":"42'10069","reported_seq":10355,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.627614+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:13.627614+0000","last_peered":"2024-09-10T20:28:13.627614+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:13.627614+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16818176,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":198}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10115","reported_seq":10251,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.508439+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:11.508439+0000","last_peered":"2024-09-10T20:28:11.508439+0000","last_clean":"2024-09-10T20:28:11.508439+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:11.508439+0000","last_undegraded":"2024-09-10T20:28:11.508439+0000","last_fullsized":"2024-09-10T20:28:11.508439+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17506304,"num_objects":2137,"num_object_clones":0,"num_object_copies":8548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10030","reported_seq":10238,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:10.082633+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:10.082633+0000","last_peered":"2024-09-10T20:28:10.082633+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:10.082633+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17283688,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10029,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10272","reported_seq":14446,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.760399+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:11.760399+0000","last_peered":"2024-09-10T20:28:11.760399+0000","last_clean":"2024-09-10T20:28:11.760399+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:11.760399+0000","last_undegraded":"2024-09-10T20:28:11.760399+0000","last_fullsized":"2024-09-10T20:28:11.760399+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17481728,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4532,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10120","reported_seq":10445,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.713281+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:13.713281+0000","last_peered":"2024-09-10T20:28:13.713281+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:13.713281+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782496,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10085","reported_seq":10309,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.744470+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:13.744470+0000","last_peered":"2024-09-10T20:28:13.744470+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:13.744470+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16490496,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10085,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10041","reported_seq":10249,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:12.457749+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:12.457749+0000","last_peered":"2024-09-10T20:28:12.457749+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:12.457749+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16618656,"num_objects":2029,"num_object_clones":0,"num_object_copies":8116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10185","reported_seq":10358,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:12.391220+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:12.391220+0000","last_peered":"2024-09-10T20:28:12.391220+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:12.391220+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17265824,"num_objects":2108,"num_object_clones":0,"num_object_copies":8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9943","reported_seq":13920,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:13.685455+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:13.685455+0000","last_peered":"2024-09-10T20:28:13.685455+0000","last_clean":"2024-09-10T20:28:13.685455+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:13.685455+0000","last_undegraded":"2024-09-10T20:28:13.685455+0000","last_fullsized":"2024-09-10T20:28:13.685455+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16857680,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6496,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9946","reported_seq":13921,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:10.077015+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:10.077015+0000","last_peered":"2024-09-10T20:28:10.077015+0000","last_clean":"2024-09-10T20:28:10.077015+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:10.077015+0000","last_undegraded":"2024-09-10T20:28:10.077015+0000","last_fullsized":"2024-09-10T20:28:10.077015+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17989632,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10014","reported_seq":10291,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.717056+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:13.717056+0000","last_peered":"2024-09-10T20:28:13.717056+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:13.717056+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17190640,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10011,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10100","reported_seq":10402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.260922+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:13.260922+0000","last_peered":"2024-09-10T20:28:13.260922+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:13.260922+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16915024,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9742","reported_seq":9930,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:09.957658+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:09.957658+0000","last_peered":"2024-09-10T20:28:09.957658+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:09.957658+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111632,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10048","reported_seq":11790,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.726873+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:13.726873+0000","last_peered":"2024-09-10T20:28:13.726873+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:13.726873+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16757920,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10046,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":903}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10268","reported_seq":10478,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:10.118251+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:10.118251+0000","last_peered":"2024-09-10T20:28:10.118251+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:10.118251+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9999","reported_seq":14099,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.579825+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:11.579825+0000","last_peered":"2024-09-10T20:28:11.579825+0000","last_clean":"2024-09-10T20:28:11.579825+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:11.579825+0000","last_undegraded":"2024-09-10T20:28:11.579825+0000","last_fullsized":"2024-09-10T20:28:11.579825+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17399808,"num_objects":2124,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":872,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":274333544,"num_objects":33492,"num_object_clones":0,"num_object_copies":133968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131726,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":549535744,"data_stored":549535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160277,"ondisk_log_size":160277,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725052,"kb_used_data":133860,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93646788,"statfs":{"total":96636764160,"available":95894310912,"internally_reserved":0,"allocated":137072640,"data_stored":136850124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731708,"kb_used_data":136876,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93640132,"statfs":{"total":96636764160,"available":95887495168,"internally_reserved":0,"allocated":140161024,"data_stored":139938508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599784,"kb_used_data":134924,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93772056,"statfs":{"total":96636764160,"available":96022585344,"internally_reserved":0,"allocated":138162176,"data_stored":137939660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729764,"kb_used_data":134484,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93642076,"statfs":{"total":96636764160,"available":95889485824,"internally_reserved":0,"allocated":137711616,"data_stored":137489100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,6,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136974336,"data_stored":136974336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137707520,"data_stored":137707520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138993664,"data_stored":138993664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135860224,"data_stored":135860224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:14.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:14.711 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-10T20:28:14.071213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274333544,"num_objects":33492,"num_object_clones":0,"num_object_copies":133968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131726,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160277,"ondisk_log_size":160277,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786308,"kb_used_data":540144,"kb_used_omap":14,"kb_used_meta":2246065,"kb_avail":374701052,"statfs":{"total":386547056640,"available":383693877248,"internally_reserved":0,"allocated":553107456,"data_stored":552217392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2299971126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,4,21,25,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22233088,"num_objects":-2714,"num_object_clones":0,"num_object_copies":-10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004674"},"pg_stats":[{"pgid":"1.f","version":"42'10069","reported_seq":10355,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.627614+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:13.627614+0000","last_peered":"2024-09-10T20:28:13.627614+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:13.627614+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16818176,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":198}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10115","reported_seq":10251,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.508439+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:11.508439+0000","last_peered":"2024-09-10T20:28:11.508439+0000","last_clean":"2024-09-10T20:28:11.508439+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:11.508439+0000","last_undegraded":"2024-09-10T20:28:11.508439+0000","last_fullsized":"2024-09-10T20:28:11.508439+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17506304,"num_objects":2137,"num_object_clones":0,"num_object_copies":8548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10030","reported_seq":10238,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:10.082633+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:10.082633+0000","last_peered":"2024-09-10T20:28:10.082633+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:10.082633+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17283688,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10029,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10272","reported_seq":14446,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.760399+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:11.760399+0000","last_peered":"2024-09-10T20:28:11.760399+0000","last_clean":"2024-09-10T20:28:11.760399+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:11.760399+0000","last_undegraded":"2024-09-10T20:28:11.760399+0000","last_fullsized":"2024-09-10T20:28:11.760399+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17481728,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4532,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10120","reported_seq":10445,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.713281+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:13.713281+0000","last_peered":"2024-09-10T20:28:13.713281+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:13.713281+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782496,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10085","reported_seq":10309,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.744470+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:13.744470+0000","last_peered":"2024-09-10T20:28:13.744470+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:13.744470+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16490496,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10085,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10041","reported_seq":10249,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:12.457749+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:12.457749+0000","last_peered":"2024-09-10T20:28:12.457749+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:12.457749+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16618656,"num_objects":2029,"num_object_clones":0,"num_object_copies":8116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10185","reported_seq":10358,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:12.391220+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:12.391220+0000","last_peered":"2024-09-10T20:28:12.391220+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:12.391220+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17265824,"num_objects":2108,"num_object_clones":0,"num_object_copies":8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9943","reported_seq":13920,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:13.685455+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:13.685455+0000","last_peered":"2024-09-10T20:28:13.685455+0000","last_clean":"2024-09-10T20:28:13.685455+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:13.685455+0000","last_undegraded":"2024-09-10T20:28:13.685455+0000","last_fullsized":"2024-09-10T20:28:13.685455+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16857680,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6496,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9946","reported_seq":13921,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:10.077015+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:10.077015+0000","last_peered":"2024-09-10T20:28:10.077015+0000","last_clean":"2024-09-10T20:28:10.077015+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:10.077015+0000","last_undegraded":"2024-09-10T20:28:10.077015+0000","last_fullsized":"2024-09-10T20:28:10.077015+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17989632,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10014","reported_seq":10291,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.717056+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:13.717056+0000","last_peered":"2024-09-10T20:28:13.717056+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:13.717056+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17190640,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10011,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10100","reported_seq":10402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.260922+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:13.260922+0000","last_peered":"2024-09-10T20:28:13.260922+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:13.260922+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16915024,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9742","reported_seq":9930,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:09.957658+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:09.957658+0000","last_peered":"2024-09-10T20:28:09.957658+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:09.957658+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111632,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10048","reported_seq":11790,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.726873+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:13.726873+0000","last_peered":"2024-09-10T20:28:13.726873+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:13.726873+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16757920,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10046,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":903}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10268","reported_seq":10478,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:10.118251+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:10.118251+0000","last_peered":"2024-09-10T20:28:10.118251+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:10.118251+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9999","reported_seq":14099,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.579825+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:11.579825+0000","last_peered":"2024-09-10T20:28:11.579825+0000","last_clean":"2024-09-10T20:28:11.579825+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:11.579825+0000","last_undegraded":"2024-09-10T20:28:11.579825+0000","last_fullsized":"2024-09-10T20:28:11.579825+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17399808,"num_objects":2124,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":872,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":274333544,"num_objects":33492,"num_object_clones":0,"num_object_copies":133968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131726,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":549535744,"data_stored":549535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160277,"ondisk_log_size":160277,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725052,"kb_used_data":133860,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93646788,"statfs":{"total":96636764160,"available":95894310912,"internally_reserved":0,"allocated":137072640,"data_stored":136850124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731708,"kb_used_data":136876,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93640132,"statfs":{"total":96636764160,"available":95887495168,"internally_reserved":0,"allocated":140161024,"data_stored":139938508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599784,"kb_used_data":134924,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93772056,"statfs":{"total":96636764160,"available":96022585344,"internally_reserved":0,"allocated":138162176,"data_stored":137939660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729764,"kb_used_data":134484,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93642076,"statfs":{"total":96636764160,"available":95889485824,"internally_reserved":0,"allocated":137711616,"data_stored":137489100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,6,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136974336,"data_stored":136974336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137707520,"data_stored":137707520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138993664,"data_stored":138993664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135860224,"data_stored":135860224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10069", 'reported_seq': 10355, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:13.627614+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:13.627614+0000', 'last_peered': '2024-09-10T20:28:13.627614+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:13.627614+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16818176, 'num_objects': 2053, 'num_object_clones': 0, 'num_object_copies': 8212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10069, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 198}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10030", 'reported_seq': 10238, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:10.082633+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:10.082633+0000', 'last_peered': '2024-09-10T20:28:10.082633+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:10.082633+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17283688, 'num_objects': 2111, 'num_object_clones': 0, 'num_object_copies': 8444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2111, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10029, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 203}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10120", 'reported_seq': 10445, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:13.713281+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:13.713281+0000', 'last_peered': '2024-09-10T20:28:13.713281+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:13.713281+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16782496, 'num_objects': 2049, 'num_object_clones': 0, 'num_object_copies': 8196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10118, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 194}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10085", 'reported_seq': 10309, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:13.744470+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:13.744470+0000', 'last_peered': '2024-09-10T20:28:13.744470+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:13.744470+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16490496, 'num_objects': 2013, 'num_object_clones': 0, 'num_object_copies': 8052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10085, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 156}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:14.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10041", 'reported_seq': 10249, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:12.457749+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:12.457749+0000', 'last_peered': '2024-09-10T20:28:12.457749+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:12.457749+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16618656, 'num_objects': 2029, 'num_object_clones': 0, 'num_object_copies': 8116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10039, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 136}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10185", 'reported_seq': 10358, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:12.391220+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:12.391220+0000', 'last_peered': '2024-09-10T20:28:12.391220+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:12.391220+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17265824, 'num_objects': 2108, 'num_object_clones': 0, 'num_object_copies': 8432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10183, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10014", 'reported_seq': 10291, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:13.717056+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:13.717056+0000', 'last_peered': '2024-09-10T20:28:13.717056+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:13.717056+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17190640, 'num_objects': 2099, 'num_object_clones': 0, 'num_object_copies': 8396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10011, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 243}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10100", 'reported_seq': 10402, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:13.260922+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:13.260922+0000', 'last_peered': '2024-09-10T20:28:13.260922+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:13.260922+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16915024, 'num_objects': 2065, 'num_object_clones': 0, 'num_object_copies': 8260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10099, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 205}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9742", 'reported_seq': 9930, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:09.957658+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:09.957658+0000', 'last_peered': '2024-09-10T20:28:09.957658+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:09.957658+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9742, 'log_dups_size': 0, 'ondisk_log_size': 9742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17111632, 'num_objects': 2089, 'num_object_clones': 0, 'num_object_copies': 8356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9741, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:14.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10048", 'reported_seq': 11790, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:13.726873+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:13.726873+0000', 'last_peered': '2024-09-10T20:28:13.726873+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:13.726873+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16757920, 'num_objects': 2046, 'num_object_clones': 0, 'num_object_copies': 8184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10046, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 836, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 903}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:14.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:14.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10268", 'reported_seq': 10478, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:10.118251+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:10.118251+0000', 'last_peered': '2024-09-10T20:28:10.118251+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:10.118251+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17863840, 'num_objects': 2181, 'num_object_clones': 0, 'num_object_copies': 8724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10266, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:14.715 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:15.412 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:15.412 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20052/133968 objects degraded (14.968%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":584,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33492,"data_bytes":274333544,"bytes_used":2853179392,"bytes_avail":383693877248,"bytes_total":386547056640,"degraded_objects":20052,"degraded_total":133968,"degraded_ratio":0.14967753493371552,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:15.441 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:15.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:16.069 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-10T20:28:14.071213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274333544,"num_objects":33492,"num_object_clones":0,"num_object_copies":133968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131726,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160277,"ondisk_log_size":160277,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786308,"kb_used_data":540144,"kb_used_omap":14,"kb_used_meta":2246065,"kb_avail":374701052,"statfs":{"total":386547056640,"available":383693877248,"internally_reserved":0,"allocated":553107456,"data_stored":552217392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2299971126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,4,21,25,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22233088,"num_objects":-2714,"num_object_clones":0,"num_object_copies":-10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004674"},"pg_stats":[{"pgid":"1.f","version":"42'10069","reported_seq":10355,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.627614+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:13.627614+0000","last_peered":"2024-09-10T20:28:13.627614+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:13.627614+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16818176,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":198}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10115","reported_seq":10251,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.508439+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:11.508439+0000","last_peered":"2024-09-10T20:28:11.508439+0000","last_clean":"2024-09-10T20:28:11.508439+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:11.508439+0000","last_undegraded":"2024-09-10T20:28:11.508439+0000","last_fullsized":"2024-09-10T20:28:11.508439+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17506304,"num_objects":2137,"num_object_clones":0,"num_object_copies":8548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10030","reported_seq":10238,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:10.082633+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:10.082633+0000","last_peered":"2024-09-10T20:28:10.082633+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:10.082633+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17283688,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10029,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10272","reported_seq":14446,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.760399+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:11.760399+0000","last_peered":"2024-09-10T20:28:11.760399+0000","last_clean":"2024-09-10T20:28:11.760399+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:11.760399+0000","last_undegraded":"2024-09-10T20:28:11.760399+0000","last_fullsized":"2024-09-10T20:28:11.760399+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17481728,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4532,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10120","reported_seq":10445,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.713281+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:13.713281+0000","last_peered":"2024-09-10T20:28:13.713281+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:13.713281+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782496,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10085","reported_seq":10309,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.744470+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:13.744470+0000","last_peered":"2024-09-10T20:28:13.744470+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:13.744470+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16490496,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10085,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10041","reported_seq":10249,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:12.457749+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:12.457749+0000","last_peered":"2024-09-10T20:28:12.457749+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:12.457749+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16618656,"num_objects":2029,"num_object_clones":0,"num_object_copies":8116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10185","reported_seq":10358,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:12.391220+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:12.391220+0000","last_peered":"2024-09-10T20:28:12.391220+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:12.391220+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17265824,"num_objects":2108,"num_object_clones":0,"num_object_copies":8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9943","reported_seq":13920,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:13.685455+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:13.685455+0000","last_peered":"2024-09-10T20:28:13.685455+0000","last_clean":"2024-09-10T20:28:13.685455+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:13.685455+0000","last_undegraded":"2024-09-10T20:28:13.685455+0000","last_fullsized":"2024-09-10T20:28:13.685455+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16857680,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6496,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'9946","reported_seq":13921,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:10.077015+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:10.077015+0000","last_peered":"2024-09-10T20:28:10.077015+0000","last_clean":"2024-09-10T20:28:10.077015+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:10.077015+0000","last_undegraded":"2024-09-10T20:28:10.077015+0000","last_fullsized":"2024-09-10T20:28:10.077015+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17989632,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10014","reported_seq":10291,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.717056+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:13.717056+0000","last_peered":"2024-09-10T20:28:13.717056+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:13.717056+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17190640,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10011,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10100","reported_seq":10402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.260922+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:13.260922+0000","last_peered":"2024-09-10T20:28:13.260922+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:13.260922+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16915024,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9742","reported_seq":9930,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:09.957658+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:09.957658+0000","last_peered":"2024-09-10T20:28:09.957658+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:09.957658+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111632,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10048","reported_seq":11790,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.726873+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:13.726873+0000","last_peered":"2024-09-10T20:28:13.726873+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:13.726873+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16757920,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10046,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":903}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10268","reported_seq":10478,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:10.118251+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:10.118251+0000","last_peered":"2024-09-10T20:28:10.118251+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:10.118251+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'9999","reported_seq":14099,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:11.579825+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:11.579825+0000","last_peered":"2024-09-10T20:28:11.579825+0000","last_clean":"2024-09-10T20:28:11.579825+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:11.579825+0000","last_undegraded":"2024-09-10T20:28:11.579825+0000","last_fullsized":"2024-09-10T20:28:11.579825+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17399808,"num_objects":2124,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":872,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":274333544,"num_objects":33492,"num_object_clones":0,"num_object_copies":133968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131726,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":549535744,"data_stored":549535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160277,"ondisk_log_size":160277,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725052,"kb_used_data":133860,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93646788,"statfs":{"total":96636764160,"available":95894310912,"internally_reserved":0,"allocated":137072640,"data_stored":136850124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731708,"kb_used_data":136876,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93640132,"statfs":{"total":96636764160,"available":95887495168,"internally_reserved":0,"allocated":140161024,"data_stored":139938508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599784,"kb_used_data":134924,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93772056,"statfs":{"total":96636764160,"available":96022585344,"internally_reserved":0,"allocated":138162176,"data_stored":137939660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729764,"kb_used_data":134484,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93642076,"statfs":{"total":96636764160,"available":95889485824,"internally_reserved":0,"allocated":137711616,"data_stored":137489100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,6,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136974336,"data_stored":136974336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137707520,"data_stored":137707520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138993664,"data_stored":138993664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135860224,"data_stored":135860224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:19.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:19.712 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:19.716 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:19.732 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-10T20:28:18.072547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269434728,"num_objects":32894,"num_object_clones":0,"num_object_copies":131576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132324,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160575,"ondisk_log_size":160575,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785672,"kb_used_data":526992,"kb_used_omap":14,"kb_used_meta":2258545,"kb_avail":374701688,"statfs":{"total":386547056640,"available":383694528512,"internally_reserved":0,"allocated":539639808,"data_stored":538749744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2312750646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,7,18,23,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20119552,"num_objects":-2456,"num_object_clones":0,"num_object_copies":-9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2456,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004486"},"pg_stats":[{"pgid":"1.f","version":"42'10069","reported_seq":10355,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.627614+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:13.627614+0000","last_peered":"2024-09-10T20:28:13.627614+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:13.627614+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16818176,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":198}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10188","reported_seq":10323,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.877821+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:16.877821+0000","last_peered":"2024-09-10T20:28:16.877821+0000","last_clean":"2024-09-10T20:28:16.877821+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:16.877821+0000","last_undegraded":"2024-09-10T20:28:16.877821+0000","last_fullsized":"2024-09-10T20:28:16.877821+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":2064,"num_object_clones":0,"num_object_copies":8256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10093","reported_seq":10301,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.264777+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:15.264777+0000","last_peered":"2024-09-10T20:28:15.264777+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:15.264777+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16767592,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10092,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10334","reported_seq":14508,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.771358+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:16.771358+0000","last_peered":"2024-09-10T20:28:16.771358+0000","last_clean":"2024-09-10T20:28:16.771358+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:16.771358+0000","last_undegraded":"2024-09-10T20:28:16.771358+0000","last_fullsized":"2024-09-10T20:28:16.771358+0000","mapping_epoch":37,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10120","reported_seq":10445,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.713281+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:13.713281+0000","last_peered":"2024-09-10T20:28:13.713281+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:13.713281+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782496,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10085","reported_seq":10309,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.744470+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:13.744470+0000","last_peered":"2024-09-10T20:28:13.744470+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:13.744470+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16490496,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10085,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10098","reported_seq":10306,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:17.100632+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:17.100632+0000","last_peered":"2024-09-10T20:28:17.100632+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:17.100632+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16151712,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":79}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10236","reported_seq":10409,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:17.440420+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:17.440420+0000","last_peered":"2024-09-10T20:28:17.440420+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:17.440420+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848032,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":57}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'9943","reported_seq":13920,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:13.685455+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:13.685455+0000","last_peered":"2024-09-10T20:28:13.685455+0000","last_clean":"2024-09-10T20:28:13.685455+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:13.685455+0000","last_undegraded":"2024-09-10T20:28:13.685455+0000","last_fullsized":"2024-09-10T20:28:13.685455+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16857680,"num_objects":2058,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6496,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10016","reported_seq":13988,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:15.153596+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:15.153596+0000","last_peered":"2024-09-10T20:28:15.153596+0000","last_clean":"2024-09-10T20:28:15.153596+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:15.153596+0000","last_undegraded":"2024-09-10T20:28:15.153596+0000","last_fullsized":"2024-09-10T20:28:15.153596+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17416192,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10014","reported_seq":10291,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.717056+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:13.717056+0000","last_peered":"2024-09-10T20:28:13.717056+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:13.717056+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17190640,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10011,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10100","reported_seq":10402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.260922+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:13.260922+0000","last_peered":"2024-09-10T20:28:13.260922+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:13.260922+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16915024,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9799","reported_seq":9986,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.304716+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:15.304716+0000","last_peered":"2024-09-10T20:28:15.304716+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:15.304716+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16644688,"num_objects":2032,"num_object_clones":0,"num_object_copies":8128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10048","reported_seq":11790,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:13.726873+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:13.726873+0000","last_peered":"2024-09-10T20:28:13.726873+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:13.726873+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16757920,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10046,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":903}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10353","reported_seq":10563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.065227+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:15.065227+0000","last_peered":"2024-09-10T20:28:15.065227+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:15.065227+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10079","reported_seq":14179,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.853422+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:16.853422+0000","last_peered":"2024-09-10T20:28:16.853422+0000","last_clean":"2024-09-10T20:28:16.853422+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:16.853422+0000","last_undegraded":"2024-09-10T20:28:16.853422+0000","last_fullsized":"2024-09-10T20:28:16.853422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16744448,"num_objects":2044,"num_object_clones":0,"num_object_copies":8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":952,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":269434728,"num_objects":32894,"num_object_clones":0,"num_object_copies":131576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132324,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536637440,"data_stored":536637440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160575,"ondisk_log_size":160575,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725052,"kb_used_data":133860,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93646788,"statfs":{"total":96636764160,"available":95894310912,"internally_reserved":0,"allocated":137072640,"data_stored":136850124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730976,"kb_used_data":131936,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93640864,"statfs":{"total":96636764160,"available":95888244736,"internally_reserved":0,"allocated":135102464,"data_stored":134879948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599864,"kb_used_data":130856,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93771976,"statfs":{"total":96636764160,"available":96022503424,"internally_reserved":0,"allocated":133996544,"data_stored":133774028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480243410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729780,"kb_used_data":130340,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93642060,"statfs":{"total":96636764160,"available":95889469440,"internally_reserved":0,"allocated":133468160,"data_stored":133245644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132694016,"data_stored":132694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133414912,"data_stored":133414912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134668288,"data_stored":134668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135860224,"data_stored":135860224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:19.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:20.411 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:20.412 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:20.429 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-10T20:28:20.073350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":265682792,"num_objects":32436,"num_object_clones":0,"num_object_copies":129744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132782,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785692,"kb_used_data":522848,"kb_used_omap":14,"kb_used_meta":2262705,"kb_avail":374701668,"statfs":{"total":386547056640,"available":383694508032,"internally_reserved":0,"allocated":535396352,"data_stored":534506288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2317010486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,7,9,19,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21086208,"num_objects":-2574,"num_object_clones":0,"num_object_copies":-10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004468"},"pg_stats":[{"pgid":"1.f","version":"42'10142","reported_seq":10428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.798381+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:18.798381+0000","last_peered":"2024-09-10T20:28:18.798381+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:18.798381+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16220160,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10188","reported_seq":10323,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.877821+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:16.877821+0000","last_peered":"2024-09-10T20:28:16.877821+0000","last_clean":"2024-09-10T20:28:16.877821+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:16.877821+0000","last_undegraded":"2024-09-10T20:28:16.877821+0000","last_fullsized":"2024-09-10T20:28:16.877821+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":2064,"num_object_clones":0,"num_object_copies":8256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10093","reported_seq":10301,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.264777+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:15.264777+0000","last_peered":"2024-09-10T20:28:15.264777+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:15.264777+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16767592,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10092,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10334","reported_seq":14508,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.771358+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:16.771358+0000","last_peered":"2024-09-10T20:28:16.771358+0000","last_clean":"2024-09-10T20:28:16.771358+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:16.771358+0000","last_undegraded":"2024-09-10T20:28:16.771358+0000","last_fullsized":"2024-09-10T20:28:16.771358+0000","mapping_epoch":37,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10169","reported_seq":10494,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.740033+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:18.740033+0000","last_peered":"2024-09-10T20:28:18.740033+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:18.740033+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16381088,"num_objects":2000,"num_object_clones":0,"num_object_copies":8000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10167,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10148","reported_seq":10372,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.725939+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:18.725939+0000","last_peered":"2024-09-10T20:28:18.725939+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:18.725939+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":93}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10098","reported_seq":10306,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:17.100632+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:17.100632+0000","last_peered":"2024-09-10T20:28:17.100632+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:17.100632+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16151712,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":79}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10236","reported_seq":10409,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:17.440420+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:17.440420+0000","last_peered":"2024-09-10T20:28:17.440420+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:17.440420+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848032,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":57}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10016","reported_seq":13993,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:18.804001+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:18.804001+0000","last_peered":"2024-09-10T20:28:18.804001+0000","last_clean":"2024-09-10T20:28:18.804001+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:18.804001+0000","last_undegraded":"2024-09-10T20:28:18.804001+0000","last_fullsized":"2024-09-10T20:28:18.804001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16259664,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6569,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10016","reported_seq":13988,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:15.153596+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:15.153596+0000","last_peered":"2024-09-10T20:28:15.153596+0000","last_clean":"2024-09-10T20:28:15.153596+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:15.153596+0000","last_undegraded":"2024-09-10T20:28:15.153596+0000","last_fullsized":"2024-09-10T20:28:15.153596+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17416192,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10074","reported_seq":10352,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.784326+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:18.784326+0000","last_peered":"2024-09-10T20:28:18.784326+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:18.784326+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16699120,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10071,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10160","reported_seq":10461,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.733901+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:18.733901+0000","last_peered":"2024-09-10T20:28:18.733901+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:18.733901+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16423504,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":145}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9799","reported_seq":9986,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.304716+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:15.304716+0000","last_peered":"2024-09-10T20:28:15.304716+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:15.304716+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16644688,"num_objects":2032,"num_object_clones":0,"num_object_copies":8128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10128","reported_seq":11964,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.859679+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:18.859679+0000","last_peered":"2024-09-10T20:28:18.859679+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:18.859679+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16102560,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10353","reported_seq":10563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.065227+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:15.065227+0000","last_peered":"2024-09-10T20:28:15.065227+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:15.065227+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10079","reported_seq":14179,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.853422+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:16.853422+0000","last_peered":"2024-09-10T20:28:16.853422+0000","last_clean":"2024-09-10T20:28:16.853422+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:16.853422+0000","last_undegraded":"2024-09-10T20:28:16.853422+0000","last_fullsized":"2024-09-10T20:28:16.853422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16744448,"num_objects":2044,"num_object_clones":0,"num_object_copies":8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":952,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":265682792,"num_objects":32436,"num_object_clones":0,"num_object_copies":129744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132782,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532291584,"data_stored":532291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725072,"kb_used_data":129716,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93646768,"statfs":{"total":96636764160,"available":95894290432,"internally_reserved":0,"allocated":132829184,"data_stored":132606668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730976,"kb_used_data":131936,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93640864,"statfs":{"total":96636764160,"available":95888244736,"internally_reserved":0,"allocated":135102464,"data_stored":134879948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599864,"kb_used_data":130856,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93771976,"statfs":{"total":96636764160,"available":96022503424,"internally_reserved":0,"allocated":133996544,"data_stored":133774028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480243410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729780,"kb_used_data":130340,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93642060,"statfs":{"total":96636764160,"available":95889469440,"internally_reserved":0,"allocated":133468160,"data_stored":133245644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132694016,"data_stored":132694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133414912,"data_stored":133414912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134668288,"data_stored":134668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":131514368,"data_stored":131514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:20.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10142", 'reported_seq': 10428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:18.798381+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:18.798381+0000', 'last_peered': '2024-09-10T20:28:18.798381+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:18.798381+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16220160, 'num_objects': 1980, 'num_object_clones': 0, 'num_object_copies': 7920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10142, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 125}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10093", 'reported_seq': 10301, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:15.264777+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:15.264777+0000', 'last_peered': '2024-09-10T20:28:15.264777+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:15.264777+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16767592, 'num_objects': 2048, 'num_object_clones': 0, 'num_object_copies': 8192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2048, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10092, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 140}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10169", 'reported_seq': 10494, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:18.740033+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:18.740033+0000', 'last_peered': '2024-09-10T20:28:18.740033+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:18.740033+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16381088, 'num_objects': 2000, 'num_object_clones': 0, 'num_object_copies': 8000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10167, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 145}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10148", 'reported_seq': 10372, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:18.725939+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:18.725939+0000', 'last_peered': '2024-09-10T20:28:18.725939+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:18.725939+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15974400, 'num_objects': 1950, 'num_object_clones': 0, 'num_object_copies': 7800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10148, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 93}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10098", 'reported_seq': 10306, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:17.100632+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:17.100632+0000', 'last_peered': '2024-09-10T20:28:17.100632+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:17.100632+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16151712, 'num_objects': 1972, 'num_object_clones': 0, 'num_object_copies': 7888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10096, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 79}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:20.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10236", 'reported_seq': 10409, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:17.440420+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:17.440420+0000', 'last_peered': '2024-09-10T20:28:17.440420+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:17.440420+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16848032, 'num_objects': 2057, 'num_object_clones': 0, 'num_object_copies': 8228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10234, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 57}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10074", 'reported_seq': 10352, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:18.784326+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:18.784326+0000', 'last_peered': '2024-09-10T20:28:18.784326+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:18.784326+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16699120, 'num_objects': 2039, 'num_object_clones': 0, 'num_object_copies': 8156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10071, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10160", 'reported_seq': 10461, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:18.733901+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:18.733901+0000', 'last_peered': '2024-09-10T20:28:18.733901+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:18.733901+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16423504, 'num_objects': 2005, 'num_object_clones': 0, 'num_object_copies': 8020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10159, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9799", 'reported_seq': 9986, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:15.304716+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:15.304716+0000', 'last_peered': '2024-09-10T20:28:15.304716+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:15.304716+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9799, 'log_dups_size': 0, 'ondisk_log_size': 9799, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16644688, 'num_objects': 2032, 'num_object_clones': 0, 'num_object_copies': 8128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9798, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 129}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10128", 'reported_seq': 11964, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:18.859679+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:18.859679+0000', 'last_peered': '2024-09-10T20:28:18.859679+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:18.859679+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16102560, 'num_objects': 1966, 'num_object_clones': 0, 'num_object_copies': 7864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10126, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 883, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:20.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10353", 'reported_seq': 10563, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:15.065227+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:15.065227+0000', 'last_peered': '2024-09-10T20:28:15.065227+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:15.065227+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17167520, 'num_objects': 2096, 'num_object_clones': 0, 'num_object_copies': 8384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10351, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:20.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:21.151 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:21.151 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20005/129744 objects degraded (15.419%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":589,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32436,"data_bytes":265682792,"bytes_used":2852548608,"bytes_avail":383694508032,"bytes_total":386547056640,"degraded_objects":20005,"degraded_total":129744,"degraded_ratio":0.15418824762609445,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":214},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:21.167 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:21.167 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:21.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:21.762 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:21.800 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-10T20:28:20.073350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":265682792,"num_objects":32436,"num_object_clones":0,"num_object_copies":129744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132782,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785692,"kb_used_data":522848,"kb_used_omap":14,"kb_used_meta":2262705,"kb_avail":374701668,"statfs":{"total":386547056640,"available":383694508032,"internally_reserved":0,"allocated":535396352,"data_stored":534506288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2317010486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,7,9,19,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21086208,"num_objects":-2574,"num_object_clones":0,"num_object_copies":-10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004468"},"pg_stats":[{"pgid":"1.f","version":"42'10142","reported_seq":10428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.798381+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:18.798381+0000","last_peered":"2024-09-10T20:28:18.798381+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:18.798381+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16220160,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10188","reported_seq":10323,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.877821+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:16.877821+0000","last_peered":"2024-09-10T20:28:16.877821+0000","last_clean":"2024-09-10T20:28:16.877821+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:16.877821+0000","last_undegraded":"2024-09-10T20:28:16.877821+0000","last_fullsized":"2024-09-10T20:28:16.877821+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":2064,"num_object_clones":0,"num_object_copies":8256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10093","reported_seq":10301,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.264777+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:15.264777+0000","last_peered":"2024-09-10T20:28:15.264777+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:15.264777+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16767592,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10092,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10334","reported_seq":14508,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.771358+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:16.771358+0000","last_peered":"2024-09-10T20:28:16.771358+0000","last_clean":"2024-09-10T20:28:16.771358+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:16.771358+0000","last_undegraded":"2024-09-10T20:28:16.771358+0000","last_fullsized":"2024-09-10T20:28:16.771358+0000","mapping_epoch":37,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10169","reported_seq":10494,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.740033+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:18.740033+0000","last_peered":"2024-09-10T20:28:18.740033+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:18.740033+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16381088,"num_objects":2000,"num_object_clones":0,"num_object_copies":8000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10167,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10148","reported_seq":10372,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.725939+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:18.725939+0000","last_peered":"2024-09-10T20:28:18.725939+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:18.725939+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":93}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10098","reported_seq":10306,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:17.100632+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:17.100632+0000","last_peered":"2024-09-10T20:28:17.100632+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:17.100632+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16151712,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":79}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10236","reported_seq":10409,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:17.440420+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:17.440420+0000","last_peered":"2024-09-10T20:28:17.440420+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:17.440420+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848032,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":57}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10016","reported_seq":13993,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:18.804001+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:18.804001+0000","last_peered":"2024-09-10T20:28:18.804001+0000","last_clean":"2024-09-10T20:28:18.804001+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:18.804001+0000","last_undegraded":"2024-09-10T20:28:18.804001+0000","last_fullsized":"2024-09-10T20:28:18.804001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16259664,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6569,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10016","reported_seq":13988,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:15.153596+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:15.153596+0000","last_peered":"2024-09-10T20:28:15.153596+0000","last_clean":"2024-09-10T20:28:15.153596+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:15.153596+0000","last_undegraded":"2024-09-10T20:28:15.153596+0000","last_fullsized":"2024-09-10T20:28:15.153596+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17416192,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10074","reported_seq":10352,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.784326+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:18.784326+0000","last_peered":"2024-09-10T20:28:18.784326+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:18.784326+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16699120,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10071,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10160","reported_seq":10461,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.733901+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:18.733901+0000","last_peered":"2024-09-10T20:28:18.733901+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:18.733901+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16423504,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":145}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9799","reported_seq":9986,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.304716+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:15.304716+0000","last_peered":"2024-09-10T20:28:15.304716+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:15.304716+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16644688,"num_objects":2032,"num_object_clones":0,"num_object_copies":8128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10128","reported_seq":11964,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:18.859679+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:18.859679+0000","last_peered":"2024-09-10T20:28:18.859679+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:18.859679+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16102560,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10353","reported_seq":10563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:15.065227+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:15.065227+0000","last_peered":"2024-09-10T20:28:15.065227+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:15.065227+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10079","reported_seq":14179,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:16.853422+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:16.853422+0000","last_peered":"2024-09-10T20:28:16.853422+0000","last_clean":"2024-09-10T20:28:16.853422+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:16.853422+0000","last_undegraded":"2024-09-10T20:28:16.853422+0000","last_fullsized":"2024-09-10T20:28:16.853422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16744448,"num_objects":2044,"num_object_clones":0,"num_object_copies":8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":952,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":265682792,"num_objects":32436,"num_object_clones":0,"num_object_copies":129744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132782,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532291584,"data_stored":532291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725072,"kb_used_data":129716,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93646768,"statfs":{"total":96636764160,"available":95894290432,"internally_reserved":0,"allocated":132829184,"data_stored":132606668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730976,"kb_used_data":131936,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93640864,"statfs":{"total":96636764160,"available":95888244736,"internally_reserved":0,"allocated":135102464,"data_stored":134879948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599864,"kb_used_data":130856,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93771976,"statfs":{"total":96636764160,"available":96022503424,"internally_reserved":0,"allocated":133996544,"data_stored":133774028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480243410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729780,"kb_used_data":130340,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93642060,"statfs":{"total":96636764160,"available":95889469440,"internally_reserved":0,"allocated":133468160,"data_stored":133245644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132694016,"data_stored":132694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133414912,"data_stored":133414912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134668288,"data_stored":134668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":131514368,"data_stored":131514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:24.802 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:25.431 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:25.432 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:25.458 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-10T20:28:24.074981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256909160,"num_objects":31365,"num_object_clones":0,"num_object_copies":125460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133853,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767768,"kb_used_data":504944,"kb_used_omap":14,"kb_used_meta":2262705,"kb_avail":374719592,"statfs":{"total":386547056640,"available":383712862208,"internally_reserved":0,"allocated":517062656,"data_stored":516172592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2317010486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,11,10,13,9,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22323200,"num_objects":-2725,"num_object_clones":0,"num_object_copies":-10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004463"},"pg_stats":[{"pgid":"1.f","version":"42'10191","reported_seq":10477,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.836179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:23.836179+0000","last_peered":"2024-09-10T20:28:23.836179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:23.836179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15818752,"num_objects":1931,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":76}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10262","reported_seq":10396,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.997826+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:21.997826+0000","last_peered":"2024-09-10T20:28:21.997826+0000","last_clean":"2024-09-10T20:28:21.997826+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:21.997826+0000","last_undegraded":"2024-09-10T20:28:21.997826+0000","last_fullsized":"2024-09-10T20:28:21.997826+0000","mapping_epoch":35,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16302080,"num_objects":1990,"num_object_clones":0,"num_object_copies":7960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10167","reported_seq":10375,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:20.352042+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:20.352042+0000","last_peered":"2024-09-10T20:28:20.352042+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:20.352042+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16161384,"num_objects":1974,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10166,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":66}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10393","reported_seq":14567,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.683482+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:21.683482+0000","last_peered":"2024-09-10T20:28:21.683482+0000","last_clean":"2024-09-10T20:28:21.683482+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:21.683482+0000","last_undegraded":"2024-09-10T20:28:21.683482+0000","last_fullsized":"2024-09-10T20:28:21.683482+0000","mapping_epoch":37,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16490496,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4653,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10231","reported_seq":10556,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.875193+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:23.875193+0000","last_peered":"2024-09-10T20:28:23.875193+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:23.875193+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15873184,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":83}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10224","reported_seq":10450,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.926077+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:23.926077+0000","last_peered":"2024-09-10T20:28:23.926077+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:23.926077+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15351808,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10224,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10155","reported_seq":10363,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:21.876608+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:21.876608+0000","last_peered":"2024-09-10T20:28:21.876608+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:21.876608+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15684768,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":22}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10313","reported_seq":10485,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:22.488871+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:22.488871+0000","last_peered":"2024-09-10T20:28:22.488871+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:22.488871+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16217248,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10084","reported_seq":14061,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:23.845706+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:23.845706+0000","last_peered":"2024-09-10T20:28:23.845706+0000","last_clean":"2024-09-10T20:28:23.845706+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:23.845706+0000","last_undegraded":"2024-09-10T20:28:23.845706+0000","last_fullsized":"2024-09-10T20:28:23.845706+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15702608,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6637,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10082","reported_seq":14057,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:20.212721+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:20.212721+0000","last_peered":"2024-09-10T20:28:20.212721+0000","last_clean":"2024-09-10T20:28:20.212721+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:20.212721+0000","last_undegraded":"2024-09-10T20:28:20.212721+0000","last_fullsized":"2024-09-10T20:28:20.212721+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16875520,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10152","reported_seq":10430,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.597140+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:23.597140+0000","last_peered":"2024-09-10T20:28:23.597140+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:23.597140+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16060144,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10229","reported_seq":10531,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.868527+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:23.868527+0000","last_peered":"2024-09-10T20:28:23.868527+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:23.868527+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15858256,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9853","reported_seq":10041,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:19.869611+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:19.869611+0000","last_peered":"2024-09-10T20:28:19.869611+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:19.869611+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9853,"log_dups_size":0,"ondisk_log_size":9853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16202320,"num_objects":1978,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9852,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":75}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10195","reported_seq":12119,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.946594+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:23.946594+0000","last_peered":"2024-09-10T20:28:23.946594+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:23.946594+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15553696,"num_objects":1899,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10415","reported_seq":10625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:20.044189+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:20.044189+0000","last_peered":"2024-09-10T20:28:20.044189+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:20.044189+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16659616,"num_objects":2034,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10158","reported_seq":14258,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.903929+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:21.903929+0000","last_peered":"2024-09-10T20:28:21.903929+0000","last_clean":"2024-09-10T20:28:21.903929+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:21.903929+0000","last_undegraded":"2024-09-10T20:28:21.903929+0000","last_fullsized":"2024-09-10T20:28:21.903929+0000","mapping_epoch":41,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16097280,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256909160,"num_objects":31365,"num_object_clones":0,"num_object_copies":125460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133853,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":515014656,"data_stored":515014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724132,"kb_used_data":124624,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93647708,"statfs":{"total":96636764160,"available":95895252992,"internally_reserved":0,"allocated":127614976,"data_stored":127392460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714132,"kb_used_data":127600,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93657708,"statfs":{"total":96636764160,"available":95905492992,"internally_reserved":0,"allocated":130662400,"data_stored":130439884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,6,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599832,"kb_used_data":126660,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93772008,"statfs":{"total":96636764160,"available":96022536192,"internally_reserved":0,"allocated":129699840,"data_stored":129477324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484503250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729672,"kb_used_data":126060,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93642168,"statfs":{"total":96636764160,"available":95889580032,"internally_reserved":0,"allocated":129085440,"data_stored":128862924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":618065615},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128450560,"data_stored":128450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129060864,"data_stored":129060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130269184,"data_stored":130269184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127234048,"data_stored":127234048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:25.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:26.038 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:26.038 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:26.060 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-10T20:28:24.074981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256909160,"num_objects":31365,"num_object_clones":0,"num_object_copies":125460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133853,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767768,"kb_used_data":504944,"kb_used_omap":14,"kb_used_meta":2262705,"kb_avail":374719592,"statfs":{"total":386547056640,"available":383712862208,"internally_reserved":0,"allocated":517062656,"data_stored":516172592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2317010486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,11,10,13,9,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22323200,"num_objects":-2725,"num_object_clones":0,"num_object_copies":-10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004463"},"pg_stats":[{"pgid":"1.f","version":"42'10191","reported_seq":10477,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.836179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:23.836179+0000","last_peered":"2024-09-10T20:28:23.836179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:23.836179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15818752,"num_objects":1931,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":76}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10262","reported_seq":10396,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.997826+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:21.997826+0000","last_peered":"2024-09-10T20:28:21.997826+0000","last_clean":"2024-09-10T20:28:21.997826+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:21.997826+0000","last_undegraded":"2024-09-10T20:28:21.997826+0000","last_fullsized":"2024-09-10T20:28:21.997826+0000","mapping_epoch":35,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16302080,"num_objects":1990,"num_object_clones":0,"num_object_copies":7960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10167","reported_seq":10375,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:20.352042+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:20.352042+0000","last_peered":"2024-09-10T20:28:20.352042+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:20.352042+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16161384,"num_objects":1974,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10166,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":66}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10393","reported_seq":14567,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.683482+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:21.683482+0000","last_peered":"2024-09-10T20:28:21.683482+0000","last_clean":"2024-09-10T20:28:21.683482+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:21.683482+0000","last_undegraded":"2024-09-10T20:28:21.683482+0000","last_fullsized":"2024-09-10T20:28:21.683482+0000","mapping_epoch":37,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16490496,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4653,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10231","reported_seq":10556,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.875193+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:23.875193+0000","last_peered":"2024-09-10T20:28:23.875193+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:23.875193+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15873184,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":83}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10224","reported_seq":10450,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.926077+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:23.926077+0000","last_peered":"2024-09-10T20:28:23.926077+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:23.926077+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15351808,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10224,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10155","reported_seq":10363,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:21.876608+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:21.876608+0000","last_peered":"2024-09-10T20:28:21.876608+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:21.876608+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15684768,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":22}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10313","reported_seq":10485,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:22.488871+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:22.488871+0000","last_peered":"2024-09-10T20:28:22.488871+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:22.488871+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16217248,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10084","reported_seq":14061,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:23.845706+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:23.845706+0000","last_peered":"2024-09-10T20:28:23.845706+0000","last_clean":"2024-09-10T20:28:23.845706+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:23.845706+0000","last_undegraded":"2024-09-10T20:28:23.845706+0000","last_fullsized":"2024-09-10T20:28:23.845706+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15702608,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6637,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10082","reported_seq":14057,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:20.212721+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:20.212721+0000","last_peered":"2024-09-10T20:28:20.212721+0000","last_clean":"2024-09-10T20:28:20.212721+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:20.212721+0000","last_undegraded":"2024-09-10T20:28:20.212721+0000","last_fullsized":"2024-09-10T20:28:20.212721+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16875520,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10152","reported_seq":10430,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.597140+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:23.597140+0000","last_peered":"2024-09-10T20:28:23.597140+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:23.597140+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16060144,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10229","reported_seq":10531,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.868527+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:23.868527+0000","last_peered":"2024-09-10T20:28:23.868527+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:23.868527+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15858256,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9853","reported_seq":10041,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:19.869611+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:19.869611+0000","last_peered":"2024-09-10T20:28:19.869611+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:19.869611+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9853,"log_dups_size":0,"ondisk_log_size":9853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16202320,"num_objects":1978,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9852,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":75}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10195","reported_seq":12119,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.946594+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:23.946594+0000","last_peered":"2024-09-10T20:28:23.946594+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:23.946594+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15553696,"num_objects":1899,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10415","reported_seq":10625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:20.044189+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:20.044189+0000","last_peered":"2024-09-10T20:28:20.044189+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:20.044189+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16659616,"num_objects":2034,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10158","reported_seq":14258,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.903929+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:21.903929+0000","last_peered":"2024-09-10T20:28:21.903929+0000","last_clean":"2024-09-10T20:28:21.903929+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:21.903929+0000","last_undegraded":"2024-09-10T20:28:21.903929+0000","last_fullsized":"2024-09-10T20:28:21.903929+0000","mapping_epoch":41,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16097280,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256909160,"num_objects":31365,"num_object_clones":0,"num_object_copies":125460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133853,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":515014656,"data_stored":515014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724132,"kb_used_data":124624,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93647708,"statfs":{"total":96636764160,"available":95895252992,"internally_reserved":0,"allocated":127614976,"data_stored":127392460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714132,"kb_used_data":127600,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93657708,"statfs":{"total":96636764160,"available":95905492992,"internally_reserved":0,"allocated":130662400,"data_stored":130439884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,6,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599832,"kb_used_data":126660,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93772008,"statfs":{"total":96636764160,"available":96022536192,"internally_reserved":0,"allocated":129699840,"data_stored":129477324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484503250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729672,"kb_used_data":126060,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93642168,"statfs":{"total":96636764160,"available":95889580032,"internally_reserved":0,"allocated":129085440,"data_stored":128862924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":618065615},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128450560,"data_stored":128450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129060864,"data_stored":129060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130269184,"data_stored":130269184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127234048,"data_stored":127234048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10191", 'reported_seq': 10477, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:23.836179+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:23.836179+0000', 'last_peered': '2024-09-10T20:28:23.836179+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:23.836179+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15818752, 'num_objects': 1931, 'num_object_clones': 0, 'num_object_copies': 7724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10191, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 76}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10167", 'reported_seq': 10375, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:20.352042+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:20.352042+0000', 'last_peered': '2024-09-10T20:28:20.352042+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:20.352042+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16161384, 'num_objects': 1974, 'num_object_clones': 0, 'num_object_copies': 7896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1974, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10166, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 66}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10231", 'reported_seq': 10556, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:23.875193+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:23.875193+0000', 'last_peered': '2024-09-10T20:28:23.875193+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:23.875193+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15873184, 'num_objects': 1938, 'num_object_clones': 0, 'num_object_copies': 7752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10229, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 83}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:26.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10224", 'reported_seq': 10450, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:23.926077+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:23.926077+0000', 'last_peered': '2024-09-10T20:28:23.926077+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:23.926077+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15351808, 'num_objects': 1874, 'num_object_clones': 0, 'num_object_copies': 7496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10224, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 17}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10155", 'reported_seq': 10363, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:21.876608+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:21.876608+0000', 'last_peered': '2024-09-10T20:28:21.876608+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:21.876608+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15684768, 'num_objects': 1915, 'num_object_clones': 0, 'num_object_copies': 7660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10153, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 22}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10313", 'reported_seq': 10485, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:22.488871+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:22.488871+0000', 'last_peered': '2024-09-10T20:28:22.488871+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:22.488871+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16217248, 'num_objects': 1980, 'num_object_clones': 0, 'num_object_copies': 7920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10311, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -20}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10152", 'reported_seq': 10430, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:23.597140+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:23.597140+0000', 'last_peered': '2024-09-10T20:28:23.597140+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:23.597140+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16060144, 'num_objects': 1961, 'num_object_clones': 0, 'num_object_copies': 7844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10149, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 105}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10229", 'reported_seq': 10531, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:23.868527+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:23.868527+0000', 'last_peered': '2024-09-10T20:28:23.868527+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:23.868527+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15858256, 'num_objects': 1936, 'num_object_clones': 0, 'num_object_copies': 7744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10228, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 76}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9853", 'reported_seq': 10041, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:19.869611+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:19.869611+0000', 'last_peered': '2024-09-10T20:28:19.869611+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:19.869611+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9853, 'log_dups_size': 0, 'ondisk_log_size': 9853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16202320, 'num_objects': 1978, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9852, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 75}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:26.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10195", 'reported_seq': 12119, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:23.946594+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:23.946594+0000', 'last_peered': '2024-09-10T20:28:23.946594+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:23.946594+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15553696, 'num_objects': 1899, 'num_object_clones': 0, 'num_object_copies': 7596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10193, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 927, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 847}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:26.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:26.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10415", 'reported_seq': 10625, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:20.044189+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:20.044189+0000', 'last_peered': '2024-09-10T20:28:20.044189+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:20.044189+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16659616, 'num_objects': 2034, 'num_object_clones': 0, 'num_object_copies': 8136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10413, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:26.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:26.790 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:26.790 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19961/124400 objects degraded (16.046%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":595,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":31100,"data_bytes":254738280,"bytes_used":2834227200,"bytes_avail":383712829440,"bytes_total":386547056640,"degraded_objects":19961,"degraded_total":124400,"degraded_ratio":0.16045819935691319,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:26.807 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:26.808 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:27.435 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:27.454 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-10T20:28:26.075605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":254738280,"num_objects":31100,"num_object_clones":0,"num_object_copies":124400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134118,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160769,"ondisk_log_size":160769,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767800,"kb_used_data":500836,"kb_used_omap":14,"kb_used_meta":2266865,"kb_avail":374719560,"statfs":{"total":386547056640,"available":383712829440,"internally_reserved":0,"allocated":512856064,"data_stored":511966000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2321270326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,11,4,11,9,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19595264,"num_objects":-2392,"num_object_clones":0,"num_object_copies":-9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004392"},"pg_stats":[{"pgid":"1.f","version":"42'10191","reported_seq":10477,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.836179+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:23.836179+0000","last_peered":"2024-09-10T20:28:23.836179+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:23.836179+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15818752,"num_objects":1931,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":76}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10262","reported_seq":10396,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.997826+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:21.997826+0000","last_peered":"2024-09-10T20:28:21.997826+0000","last_clean":"2024-09-10T20:28:21.997826+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:21.997826+0000","last_undegraded":"2024-09-10T20:28:21.997826+0000","last_fullsized":"2024-09-10T20:28:21.997826+0000","mapping_epoch":35,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16302080,"num_objects":1990,"num_object_clones":0,"num_object_copies":7960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10238","reported_seq":10446,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.413263+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:25.413263+0000","last_peered":"2024-09-10T20:28:25.413263+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:25.413263+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15579752,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10237,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10393","reported_seq":14567,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.683482+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:21.683482+0000","last_peered":"2024-09-10T20:28:21.683482+0000","last_clean":"2024-09-10T20:28:21.683482+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:21.683482+0000","last_undegraded":"2024-09-10T20:28:21.683482+0000","last_fullsized":"2024-09-10T20:28:21.683482+0000","mapping_epoch":37,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16490496,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4653,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10231","reported_seq":10556,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.875193+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:23.875193+0000","last_peered":"2024-09-10T20:28:23.875193+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:23.875193+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15873184,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":83}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10224","reported_seq":10450,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.926077+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:23.926077+0000","last_peered":"2024-09-10T20:28:23.926077+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:23.926077+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15351808,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10224,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10155","reported_seq":10363,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:21.876608+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:21.876608+0000","last_peered":"2024-09-10T20:28:21.876608+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:21.876608+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15684768,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":22}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10313","reported_seq":10485,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:22.488871+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:22.488871+0000","last_peered":"2024-09-10T20:28:22.488871+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:22.488871+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16217248,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10084","reported_seq":14061,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:23.845706+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:23.845706+0000","last_peered":"2024-09-10T20:28:23.845706+0000","last_clean":"2024-09-10T20:28:23.845706+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:23.845706+0000","last_undegraded":"2024-09-10T20:28:23.845706+0000","last_fullsized":"2024-09-10T20:28:23.845706+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15702608,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6637,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10157","reported_seq":14132,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:25.403656+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:25.403656+0000","last_peered":"2024-09-10T20:28:25.403656+0000","last_clean":"2024-09-10T20:28:25.403656+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:25.403656+0000","last_undegraded":"2024-09-10T20:28:25.403656+0000","last_fullsized":"2024-09-10T20:28:25.403656+0000","mapping_epoch":39,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16261120,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10152","reported_seq":10430,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.597140+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:23.597140+0000","last_peered":"2024-09-10T20:28:23.597140+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:23.597140+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16060144,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10229","reported_seq":10531,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.868527+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:23.868527+0000","last_peered":"2024-09-10T20:28:23.868527+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:23.868527+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15858256,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9904","reported_seq":10092,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.305766+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:25.305766+0000","last_peered":"2024-09-10T20:28:25.305766+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:25.305766+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15784528,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":24}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10195","reported_seq":12119,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:23.946594+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:23.946594+0000","last_peered":"2024-09-10T20:28:23.946594+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:23.946594+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15553696,"num_objects":1899,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10483","reported_seq":10692,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.467355+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:25.467355+0000","last_peered":"2024-09-10T20:28:25.467355+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:25.467355+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16102560,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":58}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10158","reported_seq":14258,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:21.903929+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:21.903929+0000","last_peered":"2024-09-10T20:28:21.903929+0000","last_clean":"2024-09-10T20:28:21.903929+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:21.903929+0000","last_undegraded":"2024-09-10T20:28:21.903929+0000","last_fullsized":"2024-09-10T20:28:21.903929+0000","mapping_epoch":41,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16097280,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":254738280,"num_objects":31100,"num_object_clones":0,"num_object_copies":124400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134118,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":510672896,"data_stored":510672896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160769,"ondisk_log_size":160769,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724132,"kb_used_data":124624,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93647708,"statfs":{"total":96636764160,"available":95895252992,"internally_reserved":0,"allocated":127614976,"data_stored":127392460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714164,"kb_used_data":123492,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93657676,"statfs":{"total":96636764160,"available":95905460224,"internally_reserved":0,"allocated":126455808,"data_stored":126233292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599832,"kb_used_data":126660,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93772008,"statfs":{"total":96636764160,"available":96022536192,"internally_reserved":0,"allocated":129699840,"data_stored":129477324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484503250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729672,"kb_used_data":126060,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93642168,"statfs":{"total":96636764160,"available":95889580032,"internally_reserved":0,"allocated":129085440,"data_stored":128862924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":618065615},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128450560,"data_stored":128450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129060864,"data_stored":129060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125927424,"data_stored":125927424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127234048,"data_stored":127234048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:30.456 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:31.047 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:31.048 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:31.076 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-10T20:28:30.077080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":248258408,"num_objects":30309,"num_object_clones":0,"num_object_copies":121236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134909,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717956,"kb_used_data":488432,"kb_used_omap":14,"kb_used_meta":2229425,"kb_avail":374769404,"statfs":{"total":386547056640,"available":383763869696,"internally_reserved":0,"allocated":500154368,"data_stored":499264304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2282931766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,3,8,11,15,8,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21176320,"num_objects":-2585,"num_object_clones":0,"num_object_copies":-10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004533"},"pg_stats":[{"pgid":"1.f","version":"42'10258","reported_seq":10544,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.682131+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:28.682131+0000","last_peered":"2024-09-10T20:28:28.682131+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:28.682131+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10330","reported_seq":10465,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:27.052107+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:27.052107+0000","last_peered":"2024-09-10T20:28:27.052107+0000","last_clean":"2024-09-10T20:28:27.052107+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:27.052107+0000","last_undegraded":"2024-09-10T20:28:27.052107+0000","last_fullsized":"2024-09-10T20:28:27.052107+0000","mapping_epoch":35,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15745024,"num_objects":1922,"num_object_clones":0,"num_object_copies":7688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10238","reported_seq":10446,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.413263+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:25.413263+0000","last_peered":"2024-09-10T20:28:25.413263+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:25.413263+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15579752,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10237,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10456","reported_seq":14630,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:26.754447+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:26.754447+0000","last_peered":"2024-09-10T20:28:26.754447+0000","last_clean":"2024-09-10T20:28:26.754447+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:26.754447+0000","last_undegraded":"2024-09-10T20:28:26.754447+0000","last_fullsized":"2024-09-10T20:28:26.754447+0000","mapping_epoch":37,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10287","reported_seq":10611,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.927403+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:28.927403+0000","last_peered":"2024-09-10T20:28:28.927403+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:28.927403+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15414432,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":27}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10288","reported_seq":10514,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.952936+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:28.952936+0000","last_peered":"2024-09-10T20:28:28.952936+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:28.952936+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14827520,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10214","reported_seq":10422,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:27.515657+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:27.515657+0000","last_peered":"2024-09-10T20:28:27.515657+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:27.515657+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15201440,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-37}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10373","reported_seq":10546,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:27.448497+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:27.448497+0000","last_peered":"2024-09-10T20:28:27.448497+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:27.448497+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15725728,"num_objects":1920,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-80}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10163","reported_seq":14140,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:28.990807+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:28.990807+0000","last_peered":"2024-09-10T20:28:28.990807+0000","last_clean":"2024-09-10T20:28:28.990807+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:28.990807+0000","last_undegraded":"2024-09-10T20:28:28.990807+0000","last_fullsized":"2024-09-10T20:28:28.990807+0000","mapping_epoch":33,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15055440,"num_objects":1838,"num_object_clones":0,"num_object_copies":7352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10157","reported_seq":14132,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:25.403656+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:25.403656+0000","last_peered":"2024-09-10T20:28:25.403656+0000","last_clean":"2024-09-10T20:28:25.403656+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:25.403656+0000","last_undegraded":"2024-09-10T20:28:25.403656+0000","last_fullsized":"2024-09-10T20:28:25.403656+0000","mapping_epoch":39,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16261120,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10231","reported_seq":10509,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:29.008382+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:29.008382+0000","last_peered":"2024-09-10T20:28:29.008382+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:29.008382+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15412976,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10283","reported_seq":10586,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.894747+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:28.894747+0000","last_peered":"2024-09-10T20:28:28.894747+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:28.894747+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415888,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10282,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":22}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9904","reported_seq":10092,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.305766+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:25.305766+0000","last_peered":"2024-09-10T20:28:25.305766+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:25.305766+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15784528,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":24}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10269","reported_seq":12276,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.944864+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:28.944864+0000","last_peered":"2024-09-10T20:28:28.944864+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:28.944864+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947488,"num_objects":1825,"num_object_clones":0,"num_object_copies":7300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10483","reported_seq":10692,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.467355+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:25.467355+0000","last_peered":"2024-09-10T20:28:25.467355+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:25.467355+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16102560,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":58}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10226","reported_seq":14323,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:27.020357+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:27.020357+0000","last_peered":"2024-09-10T20:28:27.020357+0000","last_clean":"2024-09-10T20:28:27.020357+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:27.020357+0000","last_undegraded":"2024-09-10T20:28:27.020357+0000","last_fullsized":"2024-09-10T20:28:27.020357+0000","mapping_epoch":41,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15540224,"num_objects":1897,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":248258408,"num_objects":30309,"num_object_clones":0,"num_object_copies":121236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134909,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497647616,"data_stored":497647616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707588,"kb_used_data":120564,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93664252,"statfs":{"total":96636764160,"available":95912194048,"internally_reserved":0,"allocated":123457536,"data_stored":123235020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":601091792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,10,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714164,"kb_used_data":123492,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93657676,"statfs":{"total":96636764160,"available":95905460224,"internally_reserved":0,"allocated":126455808,"data_stored":126233292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583136,"kb_used_data":122440,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93788704,"statfs":{"total":96636764160,"available":96039632896,"internally_reserved":0,"allocated":125378560,"data_stored":125156044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,0,0,1,3,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713068,"kb_used_data":121936,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93658772,"statfs":{"total":96636764160,"available":95906582528,"internally_reserved":0,"allocated":124862464,"data_stored":124639948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124133376,"data_stored":124133376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124674048,"data_stored":124674048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125927424,"data_stored":125927424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122912768,"data_stored":122912768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:31.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:31.709 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:31.710 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:31.729 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-10T20:28:30.077080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":248258408,"num_objects":30309,"num_object_clones":0,"num_object_copies":121236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134909,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717956,"kb_used_data":488432,"kb_used_omap":14,"kb_used_meta":2229425,"kb_avail":374769404,"statfs":{"total":386547056640,"available":383763869696,"internally_reserved":0,"allocated":500154368,"data_stored":499264304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2282931766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,3,8,11,15,8,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21176320,"num_objects":-2585,"num_object_clones":0,"num_object_copies":-10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004533"},"pg_stats":[{"pgid":"1.f","version":"42'10258","reported_seq":10544,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.682131+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:28.682131+0000","last_peered":"2024-09-10T20:28:28.682131+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:28.682131+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10330","reported_seq":10465,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:27.052107+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:27.052107+0000","last_peered":"2024-09-10T20:28:27.052107+0000","last_clean":"2024-09-10T20:28:27.052107+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:27.052107+0000","last_undegraded":"2024-09-10T20:28:27.052107+0000","last_fullsized":"2024-09-10T20:28:27.052107+0000","mapping_epoch":35,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15745024,"num_objects":1922,"num_object_clones":0,"num_object_copies":7688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10238","reported_seq":10446,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.413263+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:25.413263+0000","last_peered":"2024-09-10T20:28:25.413263+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:25.413263+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15579752,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10237,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10456","reported_seq":14630,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:26.754447+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:26.754447+0000","last_peered":"2024-09-10T20:28:26.754447+0000","last_clean":"2024-09-10T20:28:26.754447+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:26.754447+0000","last_undegraded":"2024-09-10T20:28:26.754447+0000","last_fullsized":"2024-09-10T20:28:26.754447+0000","mapping_epoch":37,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10287","reported_seq":10611,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.927403+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:28.927403+0000","last_peered":"2024-09-10T20:28:28.927403+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:28.927403+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15414432,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":27}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10288","reported_seq":10514,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.952936+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:28.952936+0000","last_peered":"2024-09-10T20:28:28.952936+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:28.952936+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14827520,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10214","reported_seq":10422,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:27.515657+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:27.515657+0000","last_peered":"2024-09-10T20:28:27.515657+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:27.515657+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15201440,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-37}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10373","reported_seq":10546,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:27.448497+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:27.448497+0000","last_peered":"2024-09-10T20:28:27.448497+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:27.448497+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15725728,"num_objects":1920,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-80}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10163","reported_seq":14140,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:28.990807+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:28.990807+0000","last_peered":"2024-09-10T20:28:28.990807+0000","last_clean":"2024-09-10T20:28:28.990807+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:28.990807+0000","last_undegraded":"2024-09-10T20:28:28.990807+0000","last_fullsized":"2024-09-10T20:28:28.990807+0000","mapping_epoch":33,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15055440,"num_objects":1838,"num_object_clones":0,"num_object_copies":7352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10157","reported_seq":14132,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:25.403656+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:25.403656+0000","last_peered":"2024-09-10T20:28:25.403656+0000","last_clean":"2024-09-10T20:28:25.403656+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:25.403656+0000","last_undegraded":"2024-09-10T20:28:25.403656+0000","last_fullsized":"2024-09-10T20:28:25.403656+0000","mapping_epoch":39,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16261120,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10231","reported_seq":10509,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:29.008382+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:29.008382+0000","last_peered":"2024-09-10T20:28:29.008382+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:29.008382+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15412976,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10283","reported_seq":10586,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.894747+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:28.894747+0000","last_peered":"2024-09-10T20:28:28.894747+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:28.894747+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415888,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10282,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":22}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9904","reported_seq":10092,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.305766+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:25.305766+0000","last_peered":"2024-09-10T20:28:25.305766+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:25.305766+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15784528,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":24}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10269","reported_seq":12276,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.944864+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:28.944864+0000","last_peered":"2024-09-10T20:28:28.944864+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:28.944864+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947488,"num_objects":1825,"num_object_clones":0,"num_object_copies":7300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10483","reported_seq":10692,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:25.467355+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:25.467355+0000","last_peered":"2024-09-10T20:28:25.467355+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:25.467355+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16102560,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":58}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10226","reported_seq":14323,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:27.020357+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:27.020357+0000","last_peered":"2024-09-10T20:28:27.020357+0000","last_clean":"2024-09-10T20:28:27.020357+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:27.020357+0000","last_undegraded":"2024-09-10T20:28:27.020357+0000","last_fullsized":"2024-09-10T20:28:27.020357+0000","mapping_epoch":41,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15540224,"num_objects":1897,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":248258408,"num_objects":30309,"num_object_clones":0,"num_object_copies":121236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134909,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497647616,"data_stored":497647616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707588,"kb_used_data":120564,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93664252,"statfs":{"total":96636764160,"available":95912194048,"internally_reserved":0,"allocated":123457536,"data_stored":123235020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":601091792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,10,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714164,"kb_used_data":123492,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93657676,"statfs":{"total":96636764160,"available":95905460224,"internally_reserved":0,"allocated":126455808,"data_stored":126233292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583136,"kb_used_data":122440,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93788704,"statfs":{"total":96636764160,"available":96039632896,"internally_reserved":0,"allocated":125378560,"data_stored":125156044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,0,0,1,3,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713068,"kb_used_data":121936,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93658772,"statfs":{"total":96636764160,"available":95906582528,"internally_reserved":0,"allocated":124862464,"data_stored":124639948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124133376,"data_stored":124133376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124674048,"data_stored":124674048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125927424,"data_stored":125927424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122912768,"data_stored":122912768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:31.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:31.730 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10258", 'reported_seq': 10544, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:28.682131+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:28.682131+0000', 'last_peered': '2024-09-10T20:28:28.682131+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:28.682131+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15269888, 'num_objects': 1864, 'num_object_clones': 0, 'num_object_copies': 7456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10258, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 9}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:31.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:31.730 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10238", 'reported_seq': 10446, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:25.413263+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:25.413263+0000', 'last_peered': '2024-09-10T20:28:25.413263+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:25.413263+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15579752, 'num_objects': 1903, 'num_object_clones': 0, 'num_object_copies': 7612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1903, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10237, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -5}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:31.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:31.730 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10287", 'reported_seq': 10611, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:28.927403+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:28.927403+0000', 'last_peered': '2024-09-10T20:28:28.927403+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:28.927403+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15414432, 'num_objects': 1882, 'num_object_clones': 0, 'num_object_copies': 7528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10285, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 27}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10288", 'reported_seq': 10514, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:28.952936+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:28.952936+0000', 'last_peered': '2024-09-10T20:28:28.952936+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:28.952936+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14827520, 'num_objects': 1810, 'num_object_clones': 0, 'num_object_copies': 7240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10288, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -47}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10214", 'reported_seq': 10422, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:27.515657+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:27.515657+0000', 'last_peered': '2024-09-10T20:28:27.515657+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:27.515657+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15201440, 'num_objects': 1856, 'num_object_clones': 0, 'num_object_copies': 7424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10212, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -37}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10373", 'reported_seq': 10546, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:27.448497+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:27.448497+0000', 'last_peered': '2024-09-10T20:28:27.448497+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:27.448497+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15725728, 'num_objects': 1920, 'num_object_clones': 0, 'num_object_copies': 7680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10371, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -80}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10231", 'reported_seq': 10509, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:29.008382+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:29.008382+0000', 'last_peered': '2024-09-10T20:28:29.008382+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:29.008382+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15412976, 'num_objects': 1882, 'num_object_clones': 0, 'num_object_copies': 7528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10228, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 26}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10283", 'reported_seq': 10586, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:28.894747+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:28.894747+0000', 'last_peered': '2024-09-10T20:28:28.894747+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:28.894747+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15415888, 'num_objects': 1882, 'num_object_clones': 0, 'num_object_copies': 7528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10282, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 22}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:31.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'9904", 'reported_seq': 10092, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:25.305766+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:25.305766+0000', 'last_peered': '2024-09-10T20:28:25.305766+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:25.305766+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 9904, 'log_dups_size': 0, 'ondisk_log_size': 9904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15784528, 'num_objects': 1927, 'num_object_clones': 0, 'num_object_copies': 7708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9903, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 24}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:31.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:31.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10269", 'reported_seq': 12276, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:28.944864+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:28.944864+0000', 'last_peered': '2024-09-10T20:28:28.944864+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:28.944864+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14947488, 'num_objects': 1825, 'num_object_clones': 0, 'num_object_copies': 7300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1011, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10267, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 968, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:31.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:31.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10483", 'reported_seq': 10692, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:25.467355+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:25.467355+0000', 'last_peered': '2024-09-10T20:28:25.467355+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:25.467355+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16102560, 'num_objects': 1966, 'num_object_clones': 0, 'num_object_copies': 7864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10481, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 58}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:31.732 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:32.500 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:32.500 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19920/120120 objects degraded (16.583%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":601,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30030,"data_bytes":245972840,"bytes_used":2783129600,"bytes_avail":383763927040,"bytes_total":386547056640,"degraded_objects":19920,"degraded_total":120120,"degraded_ratio":0.16583416583416583,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:32.522 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:32.522 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:33.152 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:33.154 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:33.185 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-10T20:28:32.077968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245972840,"num_objects":30030,"num_object_clones":0,"num_object_copies":120120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30030,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135188,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717900,"kb_used_data":484216,"kb_used_omap":14,"kb_used_meta":2233585,"kb_avail":374769460,"statfs":{"total":386547056640,"available":383763927040,"internally_reserved":0,"allocated":495837184,"data_stored":494947120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2287191606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,2,6,13,17,12,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19709952,"num_objects":-2406,"num_object_clones":0,"num_object_copies":-9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004617"},"pg_stats":[{"pgid":"1.f","version":"42'10258","reported_seq":10544,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.682131+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:28.682131+0000","last_peered":"2024-09-10T20:28:28.682131+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:28.682131+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10330","reported_seq":10465,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:27.052107+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:27.052107+0000","last_peered":"2024-09-10T20:28:27.052107+0000","last_clean":"2024-09-10T20:28:27.052107+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:27.052107+0000","last_undegraded":"2024-09-10T20:28:27.052107+0000","last_fullsized":"2024-09-10T20:28:27.052107+0000","mapping_epoch":35,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15745024,"num_objects":1922,"num_object_clones":0,"num_object_copies":7688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10301","reported_seq":10509,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:30.469942+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:30.469942+0000","last_peered":"2024-09-10T20:28:30.469942+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:30.469942+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15063656,"num_objects":1840,"num_object_clones":0,"num_object_copies":7360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1840,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10300,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-68}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10456","reported_seq":14630,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:26.754447+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:26.754447+0000","last_peered":"2024-09-10T20:28:26.754447+0000","last_clean":"2024-09-10T20:28:26.754447+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:26.754447+0000","last_undegraded":"2024-09-10T20:28:26.754447+0000","last_fullsized":"2024-09-10T20:28:26.754447+0000","mapping_epoch":37,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10287","reported_seq":10611,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.927403+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:28.927403+0000","last_peered":"2024-09-10T20:28:28.927403+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:28.927403+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15414432,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":27}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10288","reported_seq":10514,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.952936+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:28.952936+0000","last_peered":"2024-09-10T20:28:28.952936+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:28.952936+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14827520,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10214","reported_seq":10422,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:27.515657+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:27.515657+0000","last_peered":"2024-09-10T20:28:27.515657+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:27.515657+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15201440,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-37}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10373","reported_seq":10546,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:27.448497+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:27.448497+0000","last_peered":"2024-09-10T20:28:27.448497+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:27.448497+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15725728,"num_objects":1920,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-80}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10163","reported_seq":14140,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:28.990807+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:28.990807+0000","last_peered":"2024-09-10T20:28:28.990807+0000","last_clean":"2024-09-10T20:28:28.990807+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:28.990807+0000","last_undegraded":"2024-09-10T20:28:28.990807+0000","last_fullsized":"2024-09-10T20:28:28.990807+0000","mapping_epoch":33,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15055440,"num_objects":1838,"num_object_clones":0,"num_object_copies":7352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10219","reported_seq":14194,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:30.492646+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:30.492646+0000","last_peered":"2024-09-10T20:28:30.492646+0000","last_clean":"2024-09-10T20:28:30.492646+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:30.492646+0000","last_undegraded":"2024-09-10T20:28:30.492646+0000","last_fullsized":"2024-09-10T20:28:30.492646+0000","mapping_epoch":39,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15753216,"num_objects":1923,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10231","reported_seq":10509,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:29.008382+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:29.008382+0000","last_peered":"2024-09-10T20:28:29.008382+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:29.008382+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15412976,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10283","reported_seq":10586,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.894747+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:28.894747+0000","last_peered":"2024-09-10T20:28:28.894747+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:28.894747+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415888,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10282,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":22}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'9976","reported_seq":10164,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:30.435460+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:30.435460+0000","last_peered":"2024-09-10T20:28:30.435460+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:30.435460+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":9976,"log_dups_size":0,"ondisk_log_size":9976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15194704,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9975,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-48}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10269","reported_seq":12276,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:28.944864+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:28.944864+0000","last_peered":"2024-09-10T20:28:28.944864+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:28.944864+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947488,"num_objects":1825,"num_object_clones":0,"num_object_copies":7300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10565","reported_seq":10775,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:30.489265+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:30.489265+0000","last_peered":"2024-09-10T20:28:30.489265+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:30.489265+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10563,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-24}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10226","reported_seq":14323,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:27.020357+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:27.020357+0000","last_peered":"2024-09-10T20:28:27.020357+0000","last_clean":"2024-09-10T20:28:27.020357+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:27.020357+0000","last_undegraded":"2024-09-10T20:28:27.020357+0000","last_fullsized":"2024-09-10T20:28:27.020357+0000","mapping_epoch":41,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15540224,"num_objects":1897,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":245972840,"num_objects":30030,"num_object_clones":0,"num_object_copies":120120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30030,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135188,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":493264896,"data_stored":493264896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707588,"kb_used_data":120564,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93664252,"statfs":{"total":96636764160,"available":95912194048,"internally_reserved":0,"allocated":123457536,"data_stored":123235020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":601091792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,10,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714108,"kb_used_data":119276,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93657732,"statfs":{"total":96636764160,"available":95905517568,"internally_reserved":0,"allocated":122138624,"data_stored":121916108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583136,"kb_used_data":122440,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93788704,"statfs":{"total":96636764160,"available":96039632896,"internally_reserved":0,"allocated":125378560,"data_stored":125156044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,0,0,1,3,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713068,"kb_used_data":121936,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93658772,"statfs":{"total":96636764160,"available":95906582528,"internally_reserved":0,"allocated":124862464,"data_stored":124639948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124133376,"data_stored":124133376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124674048,"data_stored":124674048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":121544704,"data_stored":121544704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122912768,"data_stored":122912768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:36.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:36.812 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:36.812 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:36.842 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-10T20:28:36.079304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237559656,"num_objects":29003,"num_object_clones":0,"num_object_copies":116012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136215,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717836,"kb_used_data":467488,"kb_used_omap":14,"kb_used_meta":2250225,"kb_avail":374769524,"statfs":{"total":386547056640,"available":383763992576,"internally_reserved":0,"allocated":478707712,"data_stored":477817648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2304230966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,10,6,21,13,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19349504,"num_objects":-2362,"num_object_clones":0,"num_object_copies":-9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004322"},"pg_stats":[{"pgid":"1.f","version":"42'10324","reported_seq":10610,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.047702+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:34.047702+0000","last_peered":"2024-09-10T20:28:34.047702+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:34.047702+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14729216,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10385","reported_seq":10522,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:31.845298+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:31.845298+0000","last_peered":"2024-09-10T20:28:31.845298+0000","last_clean":"2024-09-10T20:28:31.845298+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:31.845298+0000","last_undegraded":"2024-09-10T20:28:31.845298+0000","last_fullsized":"2024-09-10T20:28:31.845298+0000","mapping_epoch":35,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15294464,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10358","reported_seq":10566,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.422928+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:35.422928+0000","last_peered":"2024-09-10T20:28:35.422928+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:35.422928+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14596712,"num_objects":1783,"num_object_clones":0,"num_object_copies":7132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10357,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-125}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10512","reported_seq":14686,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:31.979849+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:31.979849+0000","last_peered":"2024-09-10T20:28:31.979849+0000","last_clean":"2024-09-10T20:28:31.979849+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:31.979849+0000","last_undegraded":"2024-09-10T20:28:31.979849+0000","last_fullsized":"2024-09-10T20:28:31.979849+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15515648,"num_objects":1894,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10363","reported_seq":10688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.890907+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:33.890907+0000","last_peered":"2024-09-10T20:28:33.890907+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:33.890907+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14791840,"num_objects":1806,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10361,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10349","reported_seq":10576,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.853053+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:33.853053+0000","last_peered":"2024-09-10T20:28:33.853053+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:33.853053+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14327808,"num_objects":1749,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10349,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10276","reported_seq":10484,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:32.438425+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:32.438425+0000","last_peered":"2024-09-10T20:28:32.438425+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:32.438425+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14693536,"num_objects":1794,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-99}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10443","reported_seq":10616,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:32.443598+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:32.443598+0000","last_peered":"2024-09-10T20:28:32.443598+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:32.443598+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15152288,"num_objects":1850,"num_object_clones":0,"num_object_copies":7400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10230","reported_seq":14207,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:34.008416+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:34.008416+0000","last_peered":"2024-09-10T20:28:34.008416+0000","last_clean":"2024-09-10T20:28:34.008416+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:34.008416+0000","last_undegraded":"2024-09-10T20:28:34.008416+0000","last_fullsized":"2024-09-10T20:28:34.008416+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14506576,"num_objects":1771,"num_object_clones":0,"num_object_copies":7084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6783,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10294","reported_seq":14269,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:35.347128+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:35.347128+0000","last_peered":"2024-09-10T20:28:35.347128+0000","last_clean":"2024-09-10T20:28:35.347128+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:35.347128+0000","last_undegraded":"2024-09-10T20:28:35.347128+0000","last_fullsized":"2024-09-10T20:28:35.347128+0000","mapping_epoch":39,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10292","reported_seq":10570,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.971895+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:33.971895+0000","last_peered":"2024-09-10T20:28:33.971895+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:33.971895+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14913264,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-35}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10347","reported_seq":10648,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.031416+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:34.031416+0000","last_peered":"2024-09-10T20:28:34.031416+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:34.031416+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14891600,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10056","reported_seq":10244,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.410101+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:35.410101+0000","last_peered":"2024-09-10T20:28:35.410101+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:35.410101+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14539344,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10317","reported_seq":12422,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.041262+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:34.041262+0000","last_peered":"2024-09-10T20:28:34.041262+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:34.041262+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14554272,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10629","reported_seq":10839,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.440113+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:35.440113+0000","last_peered":"2024-09-10T20:28:35.440113+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:35.440113+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14906528,"num_objects":1820,"num_object_clones":0,"num_object_copies":7280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10627,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-88}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10291","reported_seq":14389,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:32.047472+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:32.047472+0000","last_peered":"2024-09-10T20:28:32.047472+0000","last_clean":"2024-09-10T20:28:32.047472+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:32.047472+0000","last_undegraded":"2024-09-10T20:28:32.047472+0000","last_fullsized":"2024-09-10T20:28:32.047472+0000","mapping_epoch":41,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":237559656,"num_objects":29003,"num_object_clones":0,"num_object_copies":116012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136215,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476098560,"data_stored":476098560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707564,"kb_used_data":116368,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93664276,"statfs":{"total":96636764160,"available":95912218624,"internally_reserved":0,"allocated":119160832,"data_stored":118938316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,0,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714236,"kb_used_data":115240,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93657604,"statfs":{"total":96636764160,"available":95905386496,"internally_reserved":0,"allocated":118005760,"data_stored":117783244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583036,"kb_used_data":118172,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93788804,"statfs":{"total":96636764160,"available":96039735296,"internally_reserved":0,"allocated":121008128,"data_stored":120785612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,4,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713000,"kb_used_data":117708,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93658840,"statfs":{"total":96636764160,"available":95906652160,"internally_reserved":0,"allocated":120532992,"data_stored":120310476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119787520,"data_stored":119787520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120307712,"data_stored":120307712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117362688,"data_stored":117362688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":118640640,"data_stored":118640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:36.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:37.413 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:37.413 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:37.437 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-10T20:28:36.079304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237559656,"num_objects":29003,"num_object_clones":0,"num_object_copies":116012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136215,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717836,"kb_used_data":467488,"kb_used_omap":14,"kb_used_meta":2250225,"kb_avail":374769524,"statfs":{"total":386547056640,"available":383763992576,"internally_reserved":0,"allocated":478707712,"data_stored":477817648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2304230966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,10,6,21,13,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19349504,"num_objects":-2362,"num_object_clones":0,"num_object_copies":-9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004322"},"pg_stats":[{"pgid":"1.f","version":"42'10324","reported_seq":10610,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.047702+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:34.047702+0000","last_peered":"2024-09-10T20:28:34.047702+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:34.047702+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14729216,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10385","reported_seq":10522,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:31.845298+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:31.845298+0000","last_peered":"2024-09-10T20:28:31.845298+0000","last_clean":"2024-09-10T20:28:31.845298+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:31.845298+0000","last_undegraded":"2024-09-10T20:28:31.845298+0000","last_fullsized":"2024-09-10T20:28:31.845298+0000","mapping_epoch":35,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15294464,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10358","reported_seq":10566,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.422928+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:35.422928+0000","last_peered":"2024-09-10T20:28:35.422928+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:35.422928+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14596712,"num_objects":1783,"num_object_clones":0,"num_object_copies":7132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10357,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-125}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10512","reported_seq":14686,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:31.979849+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:31.979849+0000","last_peered":"2024-09-10T20:28:31.979849+0000","last_clean":"2024-09-10T20:28:31.979849+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:31.979849+0000","last_undegraded":"2024-09-10T20:28:31.979849+0000","last_fullsized":"2024-09-10T20:28:31.979849+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15515648,"num_objects":1894,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10363","reported_seq":10688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.890907+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:33.890907+0000","last_peered":"2024-09-10T20:28:33.890907+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:33.890907+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14791840,"num_objects":1806,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10361,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10349","reported_seq":10576,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.853053+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:33.853053+0000","last_peered":"2024-09-10T20:28:33.853053+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:33.853053+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14327808,"num_objects":1749,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10349,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10276","reported_seq":10484,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:32.438425+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:32.438425+0000","last_peered":"2024-09-10T20:28:32.438425+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:32.438425+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14693536,"num_objects":1794,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-99}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10443","reported_seq":10616,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:32.443598+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:32.443598+0000","last_peered":"2024-09-10T20:28:32.443598+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:32.443598+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15152288,"num_objects":1850,"num_object_clones":0,"num_object_copies":7400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10230","reported_seq":14207,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:34.008416+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:34.008416+0000","last_peered":"2024-09-10T20:28:34.008416+0000","last_clean":"2024-09-10T20:28:34.008416+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:34.008416+0000","last_undegraded":"2024-09-10T20:28:34.008416+0000","last_fullsized":"2024-09-10T20:28:34.008416+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14506576,"num_objects":1771,"num_object_clones":0,"num_object_copies":7084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6783,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10294","reported_seq":14269,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:35.347128+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:35.347128+0000","last_peered":"2024-09-10T20:28:35.347128+0000","last_clean":"2024-09-10T20:28:35.347128+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:35.347128+0000","last_undegraded":"2024-09-10T20:28:35.347128+0000","last_fullsized":"2024-09-10T20:28:35.347128+0000","mapping_epoch":39,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10292","reported_seq":10570,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.971895+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:33.971895+0000","last_peered":"2024-09-10T20:28:33.971895+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:33.971895+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14913264,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-35}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10347","reported_seq":10648,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.031416+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:34.031416+0000","last_peered":"2024-09-10T20:28:34.031416+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:34.031416+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14891600,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10056","reported_seq":10244,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.410101+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:35.410101+0000","last_peered":"2024-09-10T20:28:35.410101+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:35.410101+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14539344,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10317","reported_seq":12422,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.041262+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:34.041262+0000","last_peered":"2024-09-10T20:28:34.041262+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:34.041262+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14554272,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10629","reported_seq":10839,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.440113+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:35.440113+0000","last_peered":"2024-09-10T20:28:35.440113+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:35.440113+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14906528,"num_objects":1820,"num_object_clones":0,"num_object_copies":7280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10627,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-88}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10291","reported_seq":14389,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:32.047472+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:32.047472+0000","last_peered":"2024-09-10T20:28:32.047472+0000","last_clean":"2024-09-10T20:28:32.047472+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:32.047472+0000","last_undegraded":"2024-09-10T20:28:32.047472+0000","last_fullsized":"2024-09-10T20:28:32.047472+0000","mapping_epoch":41,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":237559656,"num_objects":29003,"num_object_clones":0,"num_object_copies":116012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136215,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476098560,"data_stored":476098560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707564,"kb_used_data":116368,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93664276,"statfs":{"total":96636764160,"available":95912218624,"internally_reserved":0,"allocated":119160832,"data_stored":118938316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,0,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714236,"kb_used_data":115240,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93657604,"statfs":{"total":96636764160,"available":95905386496,"internally_reserved":0,"allocated":118005760,"data_stored":117783244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583036,"kb_used_data":118172,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93788804,"statfs":{"total":96636764160,"available":96039735296,"internally_reserved":0,"allocated":121008128,"data_stored":120785612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,4,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713000,"kb_used_data":117708,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93658840,"statfs":{"total":96636764160,"available":95906652160,"internally_reserved":0,"allocated":120532992,"data_stored":120310476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119787520,"data_stored":119787520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120307712,"data_stored":120307712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117362688,"data_stored":117362688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":118640640,"data_stored":118640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:37.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:37.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10324", 'reported_seq': 10610, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:34.047702+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:34.047702+0000', 'last_peered': '2024-09-10T20:28:34.047702+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:34.047702+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14729216, 'num_objects': 1798, 'num_object_clones': 0, 'num_object_copies': 7192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10324, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -57}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10358", 'reported_seq': 10566, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:35.422928+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:35.422928+0000', 'last_peered': '2024-09-10T20:28:35.422928+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:35.422928+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14596712, 'num_objects': 1783, 'num_object_clones': 0, 'num_object_copies': 7132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1783, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10357, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -125}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10363", 'reported_seq': 10688, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:33.890907+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:33.890907+0000', 'last_peered': '2024-09-10T20:28:33.890907+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:33.890907+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14791840, 'num_objects': 1806, 'num_object_clones': 0, 'num_object_copies': 7224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10361, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -49}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10349", 'reported_seq': 10576, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:33.853053+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:33.853053+0000', 'last_peered': '2024-09-10T20:28:33.853053+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:33.853053+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14327808, 'num_objects': 1749, 'num_object_clones': 0, 'num_object_copies': 6996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10349, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -108}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10276", 'reported_seq': 10484, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:32.438425+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:32.438425+0000', 'last_peered': '2024-09-10T20:28:32.438425+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:32.438425+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14693536, 'num_objects': 1794, 'num_object_clones': 0, 'num_object_copies': 7176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10274, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -99}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10443", 'reported_seq': 10616, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:32.443598+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:32.443598+0000', 'last_peered': '2024-09-10T20:28:32.443598+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:32.443598+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15152288, 'num_objects': 1850, 'num_object_clones': 0, 'num_object_copies': 7400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10441, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:37.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10292", 'reported_seq': 10570, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:33.971895+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:33.971895+0000', 'last_peered': '2024-09-10T20:28:33.971895+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:33.971895+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14913264, 'num_objects': 1821, 'num_object_clones': 0, 'num_object_copies': 7284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10289, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -35}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10347", 'reported_seq': 10648, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:34.031416+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:34.031416+0000', 'last_peered': '2024-09-10T20:28:34.031416+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:34.031416+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14891600, 'num_objects': 1818, 'num_object_clones': 0, 'num_object_copies': 7272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10346, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -42}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10056", 'reported_seq': 10244, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:35.410101+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:35.410101+0000', 'last_peered': '2024-09-10T20:28:35.410101+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:35.410101+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14539344, 'num_objects': 1775, 'num_object_clones': 0, 'num_object_copies': 7100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10055, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10317", 'reported_seq': 12422, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:34.041262+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:34.041262+0000', 'last_peered': '2024-09-10T20:28:34.041262+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:34.041262+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14554272, 'num_objects': 1777, 'num_object_clones': 0, 'num_object_copies': 7108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 962, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10315, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1017, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:37.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10629", 'reported_seq': 10839, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:35.440113+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:35.440113+0000', 'last_peered': '2024-09-10T20:28:35.440113+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:35.440113+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14906528, 'num_objects': 1820, 'num_object_clones': 0, 'num_object_copies': 7280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10627, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -88}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:37.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:38.245 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:38.245 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19871/116012 objects degraded (17.128%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":606,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29003,"data_bytes":237559656,"bytes_used":2783064064,"bytes_avail":383763992576,"bytes_total":386547056640,"degraded_objects":19871,"degraded_total":116012,"degraded_ratio":0.17128400510292038,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:38.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:38.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:38.941 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:38.968 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-10T20:28:38.079953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":234766184,"num_objects":28662,"num_object_clones":0,"num_object_copies":114648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136556,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735936,"kb_used_data":459244,"kb_used_omap":14,"kb_used_meta":2276593,"kb_avail":374751424,"statfs":{"total":386547056640,"available":383745458176,"internally_reserved":0,"allocated":470265856,"data_stored":469375792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2331231798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,11,4,19,18,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19972096,"num_objects":-2438,"num_object_clones":0,"num_object_copies":-9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004347"},"pg_stats":[{"pgid":"1.f","version":"42'10324","reported_seq":10610,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.047702+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:34.047702+0000","last_peered":"2024-09-10T20:28:34.047702+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:34.047702+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14729216,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10466","reported_seq":10603,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.201035+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:37.201035+0000","last_peered":"2024-09-10T20:28:37.201035+0000","last_clean":"2024-09-10T20:28:37.201035+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:37.201035+0000","last_undegraded":"2024-09-10T20:28:37.201035+0000","last_fullsized":"2024-09-10T20:28:37.201035+0000","mapping_epoch":35,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14630912,"num_objects":1786,"num_object_clones":0,"num_object_copies":7144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6907,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10358","reported_seq":10566,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.422928+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:35.422928+0000","last_peered":"2024-09-10T20:28:35.422928+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:35.422928+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14596712,"num_objects":1783,"num_object_clones":0,"num_object_copies":7132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10357,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-125}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10585","reported_seq":14759,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.175833+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:37.175833+0000","last_peered":"2024-09-10T20:28:37.175833+0000","last_clean":"2024-09-10T20:28:37.175833+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:37.175833+0000","last_undegraded":"2024-09-10T20:28:37.175833+0000","last_fullsized":"2024-09-10T20:28:37.175833+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14917632,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4845,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10363","reported_seq":10688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.890907+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:33.890907+0000","last_peered":"2024-09-10T20:28:33.890907+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:33.890907+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14791840,"num_objects":1806,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10361,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10349","reported_seq":10576,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.853053+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:33.853053+0000","last_peered":"2024-09-10T20:28:33.853053+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:33.853053+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14327808,"num_objects":1749,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10349,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10345","reported_seq":10553,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:37.493028+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:37.493028+0000","last_peered":"2024-09-10T20:28:37.493028+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:37.493028+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14128288,"num_objects":1725,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10343,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10506","reported_seq":10679,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:37.735906+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:37.735906+0000","last_peered":"2024-09-10T20:28:37.735906+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:37.735906+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14636192,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10230","reported_seq":14207,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:34.008416+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:34.008416+0000","last_peered":"2024-09-10T20:28:34.008416+0000","last_clean":"2024-09-10T20:28:34.008416+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:34.008416+0000","last_undegraded":"2024-09-10T20:28:34.008416+0000","last_fullsized":"2024-09-10T20:28:34.008416+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14506576,"num_objects":1771,"num_object_clones":0,"num_object_copies":7084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6783,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10294","reported_seq":14269,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:35.347128+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:35.347128+0000","last_peered":"2024-09-10T20:28:35.347128+0000","last_clean":"2024-09-10T20:28:35.347128+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:35.347128+0000","last_undegraded":"2024-09-10T20:28:35.347128+0000","last_fullsized":"2024-09-10T20:28:35.347128+0000","mapping_epoch":39,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10292","reported_seq":10570,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:33.971895+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:33.971895+0000","last_peered":"2024-09-10T20:28:33.971895+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:33.971895+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14913264,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-35}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10347","reported_seq":10648,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.031416+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:34.031416+0000","last_peered":"2024-09-10T20:28:34.031416+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:34.031416+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14891600,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10056","reported_seq":10244,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.410101+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:35.410101+0000","last_peered":"2024-09-10T20:28:35.410101+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:35.410101+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14539344,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10317","reported_seq":12422,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:34.041262+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:34.041262+0000","last_peered":"2024-09-10T20:28:34.041262+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:34.041262+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14554272,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10629","reported_seq":10839,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:35.440113+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:35.440113+0000","last_peered":"2024-09-10T20:28:35.440113+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:35.440113+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14906528,"num_objects":1820,"num_object_clones":0,"num_object_copies":7280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10627,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-88}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10346","reported_seq":14446,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.136220+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:37.136220+0000","last_peered":"2024-09-10T20:28:37.136220+0000","last_clean":"2024-09-10T20:28:37.136220+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:37.136220+0000","last_undegraded":"2024-09-10T20:28:37.136220+0000","last_fullsized":"2024-09-10T20:28:37.136220+0000","mapping_epoch":41,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14557184,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":234766184,"num_objects":28662,"num_object_clones":0,"num_object_copies":114648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136556,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":467595264,"data_stored":467595264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707564,"kb_used_data":116368,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93664276,"statfs":{"total":96636764160,"available":95912218624,"internally_reserved":0,"allocated":119160832,"data_stored":118938316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,0,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714236,"kb_used_data":115240,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93657604,"statfs":{"total":96636764160,"available":95905386496,"internally_reserved":0,"allocated":118005760,"data_stored":117783244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583116,"kb_used_data":114108,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93788724,"statfs":{"total":96636764160,"available":96039653376,"internally_reserved":0,"allocated":116846592,"data_stored":116624076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480243410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731020,"kb_used_data":113528,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93640820,"statfs":{"total":96636764160,"available":95888199680,"internally_reserved":0,"allocated":116252672,"data_stored":116030156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115552256,"data_stored":115552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116039680,"data_stored":116039680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117362688,"data_stored":117362688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":118640640,"data_stored":118640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:41.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:42.589 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-10T20:28:42.081374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229015400,"num_objects":27960,"num_object_clones":0,"num_object_copies":111840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137258,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772272,"kb_used_data":451180,"kb_used_omap":14,"kb_used_meta":2321009,"kb_avail":374715088,"statfs":{"total":386547056640,"available":383708250112,"internally_reserved":0,"allocated":462008320,"data_stored":461118256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2376713782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,8,20,18,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19243008,"num_objects":-2349,"num_object_clones":0,"num_object_copies":-9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2349,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004293"},"pg_stats":[{"pgid":"1.f","version":"42'10399","reported_seq":10686,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.021603+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:39.021603+0000","last_peered":"2024-09-10T20:28:39.021603+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:39.021603+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14114816,"num_objects":1723,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10399,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10466","reported_seq":10603,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.201035+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:37.201035+0000","last_peered":"2024-09-10T20:28:37.201035+0000","last_clean":"2024-09-10T20:28:37.201035+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:37.201035+0000","last_undegraded":"2024-09-10T20:28:37.201035+0000","last_fullsized":"2024-09-10T20:28:37.201035+0000","mapping_epoch":35,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14630912,"num_objects":1786,"num_object_clones":0,"num_object_copies":7144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6907,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10417","reported_seq":10625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.590216+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:40.590216+0000","last_peered":"2024-09-10T20:28:40.590216+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:40.590216+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14113384,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10416,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-184}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10585","reported_seq":14759,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.175833+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:37.175833+0000","last_peered":"2024-09-10T20:28:37.175833+0000","last_clean":"2024-09-10T20:28:37.175833+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:37.175833+0000","last_undegraded":"2024-09-10T20:28:37.175833+0000","last_fullsized":"2024-09-10T20:28:37.175833+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14917632,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4845,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10419","reported_seq":10744,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.893674+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:38.893674+0000","last_peered":"2024-09-10T20:28:38.893674+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:38.893674+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14333088,"num_objects":1750,"num_object_clones":0,"num_object_copies":7000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10413","reported_seq":10640,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.965987+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:38.965987+0000","last_peered":"2024-09-10T20:28:38.965987+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:38.965987+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13803520,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10345","reported_seq":10553,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:37.493028+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:37.493028+0000","last_peered":"2024-09-10T20:28:37.493028+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:37.493028+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14128288,"num_objects":1725,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10343,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10506","reported_seq":10679,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:37.735906+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:37.735906+0000","last_peered":"2024-09-10T20:28:37.735906+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:37.735906+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14636192,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10290","reported_seq":14267,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:39.066491+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:39.066491+0000","last_peered":"2024-09-10T20:28:39.066491+0000","last_clean":"2024-09-10T20:28:39.066491+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:39.066491+0000","last_undegraded":"2024-09-10T20:28:39.066491+0000","last_fullsized":"2024-09-10T20:28:39.066491+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14015056,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10372","reported_seq":14347,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:40.637708+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:40.637708+0000","last_peered":"2024-09-10T20:28:40.637708+0000","last_clean":"2024-09-10T20:28:40.637708+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:40.637708+0000","last_undegraded":"2024-09-10T20:28:40.637708+0000","last_fullsized":"2024-09-10T20:28:40.637708+0000","mapping_epoch":39,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14499840,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10350","reported_seq":10628,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.913049+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:38.913049+0000","last_peered":"2024-09-10T20:28:38.913049+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:38.913049+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14438128,"num_objects":1763,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10347,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-93}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10402","reported_seq":10705,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.039166+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:39.039166+0000","last_peered":"2024-09-10T20:28:39.039166+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:39.039166+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14441040,"num_objects":1763,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-97}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10119","reported_seq":10307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.632607+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:40.632607+0000","last_peered":"2024-09-10T20:28:40.632607+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:40.632607+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14023248,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10378","reported_seq":12613,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.008389+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:39.008389+0000","last_peered":"2024-09-10T20:28:39.008389+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:39.008389+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10702","reported_seq":10912,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.567997+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:40.567997+0000","last_peered":"2024-09-10T20:28:40.567997+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:40.567997+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14308512,"num_objects":1747,"num_object_clones":0,"num_object_copies":6988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10346","reported_seq":14446,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.136220+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:37.136220+0000","last_peered":"2024-09-10T20:28:37.136220+0000","last_clean":"2024-09-10T20:28:37.136220+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:37.136220+0000","last_undegraded":"2024-09-10T20:28:37.136220+0000","last_fullsized":"2024-09-10T20:28:37.136220+0000","mapping_epoch":41,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14557184,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":229015400,"num_objects":27960,"num_object_clones":0,"num_object_copies":111840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137258,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459182080,"data_stored":459182080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725716,"kb_used_data":112324,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93646124,"statfs":{"total":96636764160,"available":95893630976,"internally_reserved":0,"allocated":115019776,"data_stored":114797260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,10,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732420,"kb_used_data":111220,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93639420,"statfs":{"total":96636764160,"available":95886766080,"internally_reserved":0,"allocated":113889280,"data_stored":113666764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583116,"kb_used_data":114108,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93788724,"statfs":{"total":96636764160,"available":96039653376,"internally_reserved":0,"allocated":116846592,"data_stored":116624076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480243410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731020,"kb_used_data":113528,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93640820,"statfs":{"total":96636764160,"available":95888199680,"internally_reserved":0,"allocated":116252672,"data_stored":116030156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115552256,"data_stored":115552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116039680,"data_stored":116039680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113078272,"data_stored":113078272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114511872,"data_stored":114511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:42.590 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:43.224 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:43.225 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:43.256 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-10T20:28:42.081374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229015400,"num_objects":27960,"num_object_clones":0,"num_object_copies":111840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137258,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772272,"kb_used_data":451180,"kb_used_omap":14,"kb_used_meta":2321009,"kb_avail":374715088,"statfs":{"total":386547056640,"available":383708250112,"internally_reserved":0,"allocated":462008320,"data_stored":461118256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2376713782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,8,20,18,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19243008,"num_objects":-2349,"num_object_clones":0,"num_object_copies":-9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2349,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004293"},"pg_stats":[{"pgid":"1.f","version":"42'10399","reported_seq":10686,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.021603+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:39.021603+0000","last_peered":"2024-09-10T20:28:39.021603+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:39.021603+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14114816,"num_objects":1723,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10399,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10466","reported_seq":10603,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.201035+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:37.201035+0000","last_peered":"2024-09-10T20:28:37.201035+0000","last_clean":"2024-09-10T20:28:37.201035+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:37.201035+0000","last_undegraded":"2024-09-10T20:28:37.201035+0000","last_fullsized":"2024-09-10T20:28:37.201035+0000","mapping_epoch":35,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14630912,"num_objects":1786,"num_object_clones":0,"num_object_copies":7144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6907,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10417","reported_seq":10625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.590216+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:40.590216+0000","last_peered":"2024-09-10T20:28:40.590216+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:40.590216+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14113384,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10416,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-184}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10585","reported_seq":14759,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.175833+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:37.175833+0000","last_peered":"2024-09-10T20:28:37.175833+0000","last_clean":"2024-09-10T20:28:37.175833+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:37.175833+0000","last_undegraded":"2024-09-10T20:28:37.175833+0000","last_fullsized":"2024-09-10T20:28:37.175833+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14917632,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4845,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10419","reported_seq":10744,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.893674+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:38.893674+0000","last_peered":"2024-09-10T20:28:38.893674+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:38.893674+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14333088,"num_objects":1750,"num_object_clones":0,"num_object_copies":7000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10413","reported_seq":10640,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.965987+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:38.965987+0000","last_peered":"2024-09-10T20:28:38.965987+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:38.965987+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13803520,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10345","reported_seq":10553,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:37.493028+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:37.493028+0000","last_peered":"2024-09-10T20:28:37.493028+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:37.493028+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14128288,"num_objects":1725,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10343,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10506","reported_seq":10679,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:37.735906+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:37.735906+0000","last_peered":"2024-09-10T20:28:37.735906+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:37.735906+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14636192,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10290","reported_seq":14267,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:39.066491+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:39.066491+0000","last_peered":"2024-09-10T20:28:39.066491+0000","last_clean":"2024-09-10T20:28:39.066491+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:39.066491+0000","last_undegraded":"2024-09-10T20:28:39.066491+0000","last_fullsized":"2024-09-10T20:28:39.066491+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14015056,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10372","reported_seq":14347,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:40.637708+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:40.637708+0000","last_peered":"2024-09-10T20:28:40.637708+0000","last_clean":"2024-09-10T20:28:40.637708+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:40.637708+0000","last_undegraded":"2024-09-10T20:28:40.637708+0000","last_fullsized":"2024-09-10T20:28:40.637708+0000","mapping_epoch":39,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14499840,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10350","reported_seq":10628,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.913049+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:38.913049+0000","last_peered":"2024-09-10T20:28:38.913049+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:38.913049+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14438128,"num_objects":1763,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10347,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-93}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10402","reported_seq":10705,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.039166+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:39.039166+0000","last_peered":"2024-09-10T20:28:39.039166+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:39.039166+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14441040,"num_objects":1763,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-97}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10119","reported_seq":10307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.632607+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:40.632607+0000","last_peered":"2024-09-10T20:28:40.632607+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:40.632607+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14023248,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10378","reported_seq":12613,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.008389+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:39.008389+0000","last_peered":"2024-09-10T20:28:39.008389+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:39.008389+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10702","reported_seq":10912,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.567997+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:40.567997+0000","last_peered":"2024-09-10T20:28:40.567997+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:40.567997+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14308512,"num_objects":1747,"num_object_clones":0,"num_object_copies":6988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10346","reported_seq":14446,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:37.136220+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:37.136220+0000","last_peered":"2024-09-10T20:28:37.136220+0000","last_clean":"2024-09-10T20:28:37.136220+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:37.136220+0000","last_undegraded":"2024-09-10T20:28:37.136220+0000","last_fullsized":"2024-09-10T20:28:37.136220+0000","mapping_epoch":41,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14557184,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":229015400,"num_objects":27960,"num_object_clones":0,"num_object_copies":111840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137258,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459182080,"data_stored":459182080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725716,"kb_used_data":112324,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93646124,"statfs":{"total":96636764160,"available":95893630976,"internally_reserved":0,"allocated":115019776,"data_stored":114797260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,10,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732420,"kb_used_data":111220,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93639420,"statfs":{"total":96636764160,"available":95886766080,"internally_reserved":0,"allocated":113889280,"data_stored":113666764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583116,"kb_used_data":114108,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93788724,"statfs":{"total":96636764160,"available":96039653376,"internally_reserved":0,"allocated":116846592,"data_stored":116624076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480243410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731020,"kb_used_data":113528,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93640820,"statfs":{"total":96636764160,"available":95888199680,"internally_reserved":0,"allocated":116252672,"data_stored":116030156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115552256,"data_stored":115552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116039680,"data_stored":116039680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113078272,"data_stored":113078272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114511872,"data_stored":114511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10399", 'reported_seq': 10686, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:39.021603+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:39.021603+0000', 'last_peered': '2024-09-10T20:28:39.021603+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:39.021603+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14114816, 'num_objects': 1723, 'num_object_clones': 0, 'num_object_copies': 6892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10399, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10417", 'reported_seq': 10625, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:40.590216+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:40.590216+0000', 'last_peered': '2024-09-10T20:28:40.590216+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:40.590216+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14113384, 'num_objects': 1724, 'num_object_clones': 0, 'num_object_copies': 6896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1724, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10416, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -184}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10419", 'reported_seq': 10744, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:38.893674+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:38.893674+0000', 'last_peered': '2024-09-10T20:28:38.893674+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:38.893674+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14333088, 'num_objects': 1750, 'num_object_clones': 0, 'num_object_copies': 7000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10417, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -105}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10413", 'reported_seq': 10640, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:38.965987+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:38.965987+0000', 'last_peered': '2024-09-10T20:28:38.965987+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:38.965987+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13803520, 'num_objects': 1685, 'num_object_clones': 0, 'num_object_copies': 6740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10413, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -172}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:43.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10345", 'reported_seq': 10553, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:37.493028+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:37.493028+0000', 'last_peered': '2024-09-10T20:28:37.493028+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:37.493028+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14128288, 'num_objects': 1725, 'num_object_clones': 0, 'num_object_copies': 6900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10343, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -168}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10506", 'reported_seq': 10679, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:37.735906+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:37.735906+0000', 'last_peered': '2024-09-10T20:28:37.735906+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:37.735906+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14636192, 'num_objects': 1787, 'num_object_clones': 0, 'num_object_copies': 7148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10504, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10350", 'reported_seq': 10628, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:38.913049+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:38.913049+0000', 'last_peered': '2024-09-10T20:28:38.913049+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:38.913049+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14438128, 'num_objects': 1763, 'num_object_clones': 0, 'num_object_copies': 7052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10347, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -93}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10402", 'reported_seq': 10705, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:39.039166+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:39.039166+0000', 'last_peered': '2024-09-10T20:28:39.039166+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:39.039166+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14441040, 'num_objects': 1763, 'num_object_clones': 0, 'num_object_copies': 7052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10401, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -97}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10119", 'reported_seq': 10307, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:40.632607+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:40.632607+0000', 'last_peered': '2024-09-10T20:28:40.632607+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:40.632607+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14023248, 'num_objects': 1712, 'num_object_clones': 0, 'num_object_copies': 6848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10118, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:43.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10378", 'reported_seq': 12613, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:39.008389+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:39.008389+0000', 'last_peered': '2024-09-10T20:28:39.008389+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:39.008389+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14054560, 'num_objects': 1716, 'num_object_clones': 0, 'num_object_copies': 6864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 897, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10376, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1082, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 819}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:43.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:43.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10702", 'reported_seq': 10912, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:40.567997+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:40.567997+0000', 'last_peered': '2024-09-10T20:28:40.567997+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:40.567997+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14308512, 'num_objects': 1747, 'num_object_clones': 0, 'num_object_copies': 6988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10700, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:43.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:43.988 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:43.988 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19806/111840 objects degraded (17.709%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":612,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":27960,"data_bytes":229015400,"bytes_used":2838806528,"bytes_avail":383708250112,"bytes_total":386547056640,"degraded_objects":19806,"degraded_total":111840,"degraded_ratio":0.17709227467811156,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:44.012 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:44.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:44.663 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:44.663 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:44.691 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-10T20:28:44.082184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226287464,"num_objects":27627,"num_object_clones":0,"num_object_copies":110508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137591,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772240,"kb_used_data":442808,"kb_used_omap":14,"kb_used_meta":2329329,"kb_avail":374715120,"statfs":{"total":386547056640,"available":383708282880,"internally_reserved":0,"allocated":453435392,"data_stored":452545328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2385233462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,10,22,17,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19685376,"num_objects":-2403,"num_object_clones":0,"num_object_copies":-9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004216"},"pg_stats":[{"pgid":"1.f","version":"42'10399","reported_seq":10686,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.021603+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:39.021603+0000","last_peered":"2024-09-10T20:28:39.021603+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:39.021603+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14114816,"num_objects":1723,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10399,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10534","reported_seq":10671,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:42.086603+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:42.086603+0000","last_peered":"2024-09-10T20:28:42.086603+0000","last_clean":"2024-09-10T20:28:42.086603+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:42.086603+0000","last_undegraded":"2024-09-10T20:28:42.086603+0000","last_fullsized":"2024-09-10T20:28:42.086603+0000","mapping_epoch":35,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14073856,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10417","reported_seq":10625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.590216+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:40.590216+0000","last_peered":"2024-09-10T20:28:40.590216+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:40.590216+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14113384,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10416,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-184}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10650","reported_seq":14824,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:42.346079+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:42.346079+0000","last_peered":"2024-09-10T20:28:42.346079+0000","last_clean":"2024-09-10T20:28:42.346079+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:42.346079+0000","last_undegraded":"2024-09-10T20:28:42.346079+0000","last_fullsized":"2024-09-10T20:28:42.346079+0000","mapping_epoch":37,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14385152,"num_objects":1756,"num_object_clones":0,"num_object_copies":7024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10419","reported_seq":10744,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.893674+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:38.893674+0000","last_peered":"2024-09-10T20:28:38.893674+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:38.893674+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14333088,"num_objects":1750,"num_object_clones":0,"num_object_copies":7000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10413","reported_seq":10640,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.965987+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:38.965987+0000","last_peered":"2024-09-10T20:28:38.965987+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:38.965987+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13803520,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10416","reported_seq":10622,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:42.827446+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:42.827446+0000","last_peered":"2024-09-10T20:28:42.827446+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:42.827446+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13546656,"num_objects":1654,"num_object_clones":0,"num_object_copies":6616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-239}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10562","reported_seq":10735,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:42.512645+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:42.512645+0000","last_peered":"2024-09-10T20:28:42.512645+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:42.512645+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14177440,"num_objects":1731,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10560,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10290","reported_seq":14267,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:39.066491+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:39.066491+0000","last_peered":"2024-09-10T20:28:39.066491+0000","last_clean":"2024-09-10T20:28:39.066491+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:39.066491+0000","last_undegraded":"2024-09-10T20:28:39.066491+0000","last_fullsized":"2024-09-10T20:28:39.066491+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14015056,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10372","reported_seq":14347,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:40.637708+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:40.637708+0000","last_peered":"2024-09-10T20:28:40.637708+0000","last_clean":"2024-09-10T20:28:40.637708+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:40.637708+0000","last_undegraded":"2024-09-10T20:28:40.637708+0000","last_fullsized":"2024-09-10T20:28:40.637708+0000","mapping_epoch":39,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14499840,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10350","reported_seq":10628,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:38.913049+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:38.913049+0000","last_peered":"2024-09-10T20:28:38.913049+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:38.913049+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14438128,"num_objects":1763,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10347,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-93}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10402","reported_seq":10705,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.039166+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:39.039166+0000","last_peered":"2024-09-10T20:28:39.039166+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:39.039166+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14441040,"num_objects":1763,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-97}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10119","reported_seq":10307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.632607+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:40.632607+0000","last_peered":"2024-09-10T20:28:40.632607+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:40.632607+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14023248,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10378","reported_seq":12613,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:39.008389+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:39.008389+0000","last_peered":"2024-09-10T20:28:39.008389+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:39.008389+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14054560,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10702","reported_seq":10912,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:40.567997+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:40.567997+0000","last_peered":"2024-09-10T20:28:40.567997+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:40.567997+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14308512,"num_objects":1747,"num_object_clones":0,"num_object_copies":6988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10419","reported_seq":14519,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:42.183024+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:42.183024+0000","last_peered":"2024-09-10T20:28:42.183024+0000","last_clean":"2024-09-10T20:28:42.183024+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:42.183024+0000","last_undegraded":"2024-09-10T20:28:42.183024+0000","last_fullsized":"2024-09-10T20:28:42.183024+0000","mapping_epoch":41,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":1704,"num_object_clones":0,"num_object_copies":6816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1292,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":226287464,"num_objects":27627,"num_object_clones":0,"num_object_copies":110508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137591,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450502656,"data_stored":450502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725716,"kb_used_data":112324,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93646124,"statfs":{"total":96636764160,"available":95893630976,"internally_reserved":0,"allocated":115019776,"data_stored":114797260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,10,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732420,"kb_used_data":111220,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93639420,"statfs":{"total":96636764160,"available":95886766080,"internally_reserved":0,"allocated":113889280,"data_stored":113666764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583068,"kb_used_data":109888,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93788772,"statfs":{"total":96636764160,"available":96039702528,"internally_reserved":0,"allocated":112525312,"data_stored":112302796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484503250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215221,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731036,"kb_used_data":109376,"kb_used_omap":4,"kb_used_meta":621627,"kb_avail":93640804,"statfs":{"total":96636764160,"available":95888183296,"internally_reserved":0,"allocated":112001024,"data_stored":111778508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":636546767},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":111235072,"data_stored":111235072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111677440,"data_stored":111677440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113078272,"data_stored":113078272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114511872,"data_stored":114511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:47.693 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:48.286 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:48.312 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-10T20:28:48.083895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217857896,"num_objects":26598,"num_object_clones":0,"num_object_copies":106392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138620,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2776604,"kb_used_data":426304,"kb_used_omap":14,"kb_used_meta":2350193,"kb_avail":374710756,"statfs":{"total":386547056640,"available":383703814144,"internally_reserved":0,"allocated":436535296,"data_stored":435645232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2406598198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,5,3,7,9,15,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19701760,"num_objects":-2405,"num_object_clones":0,"num_object_copies":-9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2405,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004590"},"pg_stats":[{"pgid":"1.f","version":"42'10476","reported_seq":10763,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.093201+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:44.093201+0000","last_peered":"2024-09-10T20:28:44.093201+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:44.093201+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13484032,"num_objects":1646,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10476,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10605","reported_seq":10742,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.270427+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:47.270427+0000","last_peered":"2024-09-10T20:28:47.270427+0000","last_clean":"2024-09-10T20:28:47.270427+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:47.270427+0000","last_undegraded":"2024-09-10T20:28:47.270427+0000","last_fullsized":"2024-09-10T20:28:47.270427+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13492224,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10475","reported_seq":10683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.542245+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:45.542245+0000","last_peered":"2024-09-10T20:28:45.542245+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:45.542245+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13638248,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10474,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10719","reported_seq":14892,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.349465+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:47.349465+0000","last_peered":"2024-09-10T20:28:47.349465+0000","last_clean":"2024-09-10T20:28:47.349465+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:47.349465+0000","last_undegraded":"2024-09-10T20:28:47.349465+0000","last_fullsized":"2024-09-10T20:28:47.349465+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10485","reported_seq":10810,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:43.987514+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:43.987514+0000","last_peered":"2024-09-10T20:28:43.987514+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:43.987514+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13792416,"num_objects":1684,"num_object_clones":0,"num_object_copies":6736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10482","reported_seq":10709,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.176434+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:44.176434+0000","last_peered":"2024-09-10T20:28:44.176434+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:44.176434+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":1616,"num_object_clones":0,"num_object_copies":6464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10477","reported_seq":10685,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.755528+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:47.755528+0000","last_peered":"2024-09-10T20:28:47.755528+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:47.755528+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13046944,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10628","reported_seq":10801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.754990+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:47.754990+0000","last_peered":"2024-09-10T20:28:47.754990+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:47.754990+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13636768,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10359","reported_seq":14336,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:43.908423+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:43.908423+0000","last_peered":"2024-09-10T20:28:43.908423+0000","last_clean":"2024-09-10T20:28:43.908423+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:43.908423+0000","last_undegraded":"2024-09-10T20:28:43.908423+0000","last_fullsized":"2024-09-10T20:28:43.908423+0000","mapping_epoch":33,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13449808,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10435","reported_seq":14410,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:45.662842+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:45.662842+0000","last_peered":"2024-09-10T20:28:45.662842+0000","last_clean":"2024-09-10T20:28:45.662842+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:45.662842+0000","last_undegraded":"2024-09-10T20:28:45.662842+0000","last_fullsized":"2024-09-10T20:28:45.662842+0000","mapping_epoch":39,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13983744,"num_objects":1707,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10410","reported_seq":10688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.093797+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:44.093797+0000","last_peered":"2024-09-10T20:28:44.093797+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:44.093797+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13946608,"num_objects":1703,"num_object_clones":0,"num_object_copies":6812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10452","reported_seq":10755,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.050638+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:44.050638+0000","last_peered":"2024-09-10T20:28:44.050638+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:44.050638+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14031440,"num_objects":1713,"num_object_clones":0,"num_object_copies":6852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10451,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10183","reported_seq":10371,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.634612+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:45.634612+0000","last_peered":"2024-09-10T20:28:45.634612+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:45.634612+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13498960,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10182,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-255}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10443","reported_seq":12762,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.143493+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:44.143493+0000","last_peered":"2024-09-10T20:28:44.143493+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:44.143493+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13522080,"num_objects":1651,"num_object_clones":0,"num_object_copies":6604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":796}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10764","reported_seq":10973,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.668301+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:45.668301+0000","last_peered":"2024-09-10T20:28:45.668301+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:45.668301+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13800608,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10478","reported_seq":14576,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.335515+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:47.335515+0000","last_peered":"2024-09-10T20:28:47.335515+0000","last_clean":"2024-09-10T20:28:47.335515+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:47.335515+0000","last_undegraded":"2024-09-10T20:28:47.335515+0000","last_fullsized":"2024-09-10T20:28:47.335515+0000","mapping_epoch":41,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13475840,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":217857896,"num_objects":26598,"num_object_clones":0,"num_object_copies":106392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138620,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":433242112,"data_stored":433242112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117108,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725748,"kb_used_data":108188,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93646092,"statfs":{"total":96636764160,"available":95893598208,"internally_reserved":0,"allocated":110784512,"data_stored":110561996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,0,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733908,"kb_used_data":107140,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93637932,"statfs":{"total":96636764160,"available":95885242368,"internally_reserved":0,"allocated":109711360,"data_stored":109488844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584508,"kb_used_data":105772,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93787332,"statfs":{"total":96636764160,"available":96038227968,"internally_reserved":0,"allocated":108310528,"data_stored":108088012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732440,"kb_used_data":105204,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93639400,"statfs":{"total":96636764160,"available":95886745600,"internally_reserved":0,"allocated":107728896,"data_stored":107506380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106942464,"data_stored":106942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107388928,"data_stored":107388928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":108818432,"data_stored":108818432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110092288,"data_stored":110092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:48.313 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:48.967 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-10T20:28:48.083895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217857896,"num_objects":26598,"num_object_clones":0,"num_object_copies":106392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138620,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2776604,"kb_used_data":426304,"kb_used_omap":14,"kb_used_meta":2350193,"kb_avail":374710756,"statfs":{"total":386547056640,"available":383703814144,"internally_reserved":0,"allocated":436535296,"data_stored":435645232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2406598198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,5,3,7,9,15,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19701760,"num_objects":-2405,"num_object_clones":0,"num_object_copies":-9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2405,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004590"},"pg_stats":[{"pgid":"1.f","version":"42'10476","reported_seq":10763,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.093201+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:44.093201+0000","last_peered":"2024-09-10T20:28:44.093201+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:44.093201+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13484032,"num_objects":1646,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10476,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10605","reported_seq":10742,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.270427+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:47.270427+0000","last_peered":"2024-09-10T20:28:47.270427+0000","last_clean":"2024-09-10T20:28:47.270427+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:47.270427+0000","last_undegraded":"2024-09-10T20:28:47.270427+0000","last_fullsized":"2024-09-10T20:28:47.270427+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13492224,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10475","reported_seq":10683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.542245+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:45.542245+0000","last_peered":"2024-09-10T20:28:45.542245+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:45.542245+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13638248,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10474,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10719","reported_seq":14892,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.349465+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:47.349465+0000","last_peered":"2024-09-10T20:28:47.349465+0000","last_clean":"2024-09-10T20:28:47.349465+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:47.349465+0000","last_undegraded":"2024-09-10T20:28:47.349465+0000","last_fullsized":"2024-09-10T20:28:47.349465+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10485","reported_seq":10810,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:43.987514+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:43.987514+0000","last_peered":"2024-09-10T20:28:43.987514+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:43.987514+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13792416,"num_objects":1684,"num_object_clones":0,"num_object_copies":6736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10482","reported_seq":10709,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.176434+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:44.176434+0000","last_peered":"2024-09-10T20:28:44.176434+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:44.176434+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":1616,"num_object_clones":0,"num_object_copies":6464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10477","reported_seq":10685,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.755528+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:47.755528+0000","last_peered":"2024-09-10T20:28:47.755528+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:47.755528+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13046944,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10628","reported_seq":10801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.754990+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:47.754990+0000","last_peered":"2024-09-10T20:28:47.754990+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:47.754990+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13636768,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10359","reported_seq":14336,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:43.908423+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:43.908423+0000","last_peered":"2024-09-10T20:28:43.908423+0000","last_clean":"2024-09-10T20:28:43.908423+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:43.908423+0000","last_undegraded":"2024-09-10T20:28:43.908423+0000","last_fullsized":"2024-09-10T20:28:43.908423+0000","mapping_epoch":33,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13449808,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10435","reported_seq":14410,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:45.662842+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:45.662842+0000","last_peered":"2024-09-10T20:28:45.662842+0000","last_clean":"2024-09-10T20:28:45.662842+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:45.662842+0000","last_undegraded":"2024-09-10T20:28:45.662842+0000","last_fullsized":"2024-09-10T20:28:45.662842+0000","mapping_epoch":39,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13983744,"num_objects":1707,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10410","reported_seq":10688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.093797+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:44.093797+0000","last_peered":"2024-09-10T20:28:44.093797+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:44.093797+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13946608,"num_objects":1703,"num_object_clones":0,"num_object_copies":6812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10452","reported_seq":10755,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.050638+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:44.050638+0000","last_peered":"2024-09-10T20:28:44.050638+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:44.050638+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14031440,"num_objects":1713,"num_object_clones":0,"num_object_copies":6852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10451,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10183","reported_seq":10371,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.634612+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:45.634612+0000","last_peered":"2024-09-10T20:28:45.634612+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:45.634612+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13498960,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10182,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-255}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10443","reported_seq":12762,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:44.143493+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:44.143493+0000","last_peered":"2024-09-10T20:28:44.143493+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:44.143493+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13522080,"num_objects":1651,"num_object_clones":0,"num_object_copies":6604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":796}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10764","reported_seq":10973,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.668301+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:45.668301+0000","last_peered":"2024-09-10T20:28:45.668301+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:45.668301+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13800608,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10478","reported_seq":14576,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.335515+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:47.335515+0000","last_peered":"2024-09-10T20:28:47.335515+0000","last_clean":"2024-09-10T20:28:47.335515+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:47.335515+0000","last_undegraded":"2024-09-10T20:28:47.335515+0000","last_fullsized":"2024-09-10T20:28:47.335515+0000","mapping_epoch":41,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13475840,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":217857896,"num_objects":26598,"num_object_clones":0,"num_object_copies":106392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138620,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":433242112,"data_stored":433242112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117108,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725748,"kb_used_data":108188,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93646092,"statfs":{"total":96636764160,"available":95893598208,"internally_reserved":0,"allocated":110784512,"data_stored":110561996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,0,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733908,"kb_used_data":107140,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93637932,"statfs":{"total":96636764160,"available":95885242368,"internally_reserved":0,"allocated":109711360,"data_stored":109488844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584508,"kb_used_data":105772,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93787332,"statfs":{"total":96636764160,"available":96038227968,"internally_reserved":0,"allocated":108310528,"data_stored":108088012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732440,"kb_used_data":105204,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93639400,"statfs":{"total":96636764160,"available":95886745600,"internally_reserved":0,"allocated":107728896,"data_stored":107506380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106942464,"data_stored":106942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107388928,"data_stored":107388928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":108818432,"data_stored":108818432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110092288,"data_stored":110092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10476", 'reported_seq': 10763, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:44.093201+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:44.093201+0000', 'last_peered': '2024-09-10T20:28:44.093201+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:44.093201+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13484032, 'num_objects': 1646, 'num_object_clones': 0, 'num_object_copies': 6584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10476, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -209}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10475", 'reported_seq': 10683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:45.542245+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:45.542245+0000', 'last_peered': '2024-09-10T20:28:45.542245+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:45.542245+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13638248, 'num_objects': 1666, 'num_object_clones': 0, 'num_object_copies': 6664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1666, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10474, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -242}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10485", 'reported_seq': 10810, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:43.987514+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:43.987514+0000', 'last_peered': '2024-09-10T20:28:43.987514+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:43.987514+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13792416, 'num_objects': 1684, 'num_object_clones': 0, 'num_object_copies': 6736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10483, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -171}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10482", 'reported_seq': 10709, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:44.176434+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:44.176434+0000', 'last_peered': '2024-09-10T20:28:44.176434+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:44.176434+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13238272, 'num_objects': 1616, 'num_object_clones': 0, 'num_object_copies': 6464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10482, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -241}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:48.990 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10477", 'reported_seq': 10685, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:47.755528+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:47.755528+0000', 'last_peered': '2024-09-10T20:28:47.755528+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:47.755528+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13046944, 'num_objects': 1593, 'num_object_clones': 0, 'num_object_copies': 6372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10475, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -300}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10628", 'reported_seq': 10801, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:47.754990+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:47.754990+0000', 'last_peered': '2024-09-10T20:28:47.754990+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:47.754990+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13636768, 'num_objects': 1665, 'num_object_clones': 0, 'num_object_copies': 6660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10626, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10410", 'reported_seq': 10688, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:44.093797+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:44.093797+0000', 'last_peered': '2024-09-10T20:28:44.093797+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:44.093797+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13946608, 'num_objects': 1703, 'num_object_clones': 0, 'num_object_copies': 6812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10407, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -153}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10452", 'reported_seq': 10755, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:44.050638+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:44.050638+0000', 'last_peered': '2024-09-10T20:28:44.050638+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:44.050638+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14031440, 'num_objects': 1713, 'num_object_clones': 0, 'num_object_copies': 6852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10451, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10183", 'reported_seq': 10371, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:45.634612+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:45.634612+0000', 'last_peered': '2024-09-10T20:28:45.634612+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:45.634612+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13498960, 'num_objects': 1648, 'num_object_clones': 0, 'num_object_copies': 6592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10182, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:48.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10443", 'reported_seq': 12762, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:44.143493+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:44.143493+0000', 'last_peered': '2024-09-10T20:28:44.143493+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:44.143493+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13522080, 'num_objects': 1651, 'num_object_clones': 0, 'num_object_copies': 6604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10441, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1124, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 796}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:48.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:48.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10764", 'reported_seq': 10973, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:45.668301+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:45.668301+0000', 'last_peered': '2024-09-10T20:28:45.668301+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:45.668301+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13800608, 'num_objects': 1685, 'num_object_clones': 0, 'num_object_copies': 6740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10762, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -223}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:48.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:49.754 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19764/106392 objects degraded (18.577%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":618,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26598,"data_bytes":217857896,"bytes_used":2843242496,"bytes_avail":383703814144,"bytes_total":386547056640,"degraded_objects":19764,"degraded_total":106392,"degraded_ratio":0.18576584705616964,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:49.783 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:49.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:50.358 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:50.392 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-10T20:28:50.084692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214220648,"num_objects":26154,"num_object_clones":0,"num_object_copies":104616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26154,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139064,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778004,"kb_used_data":422132,"kb_used_omap":14,"kb_used_meta":2355761,"kb_avail":374709356,"statfs":{"total":386547056640,"available":383702380544,"internally_reserved":0,"allocated":432263168,"data_stored":431373104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2412299830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,2,8,16,13,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20545536,"num_objects":-2508,"num_object_clones":0,"num_object_copies":-10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004739"},"pg_stats":[{"pgid":"1.f","version":"42'10548","reported_seq":10835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.217832+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:49.217832+0000","last_peered":"2024-09-10T20:28:49.217832+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:49.217832+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12894208,"num_objects":1574,"num_object_clones":0,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10605","reported_seq":10742,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.270427+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:47.270427+0000","last_peered":"2024-09-10T20:28:47.270427+0000","last_clean":"2024-09-10T20:28:47.270427+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:47.270427+0000","last_undegraded":"2024-09-10T20:28:47.270427+0000","last_fullsized":"2024-09-10T20:28:47.270427+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13492224,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10475","reported_seq":10683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.542245+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:45.542245+0000","last_peered":"2024-09-10T20:28:45.542245+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:45.542245+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13638248,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10474,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10719","reported_seq":14892,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.349465+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:47.349465+0000","last_peered":"2024-09-10T20:28:47.349465+0000","last_clean":"2024-09-10T20:28:47.349465+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:47.349465+0000","last_undegraded":"2024-09-10T20:28:47.349465+0000","last_fullsized":"2024-09-10T20:28:47.349465+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10541","reported_seq":10866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.027119+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:49.027119+0000","last_peered":"2024-09-10T20:28:49.027119+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:49.027119+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13333664,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10535","reported_seq":10762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.171641+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:49.171641+0000","last_peered":"2024-09-10T20:28:49.171641+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:49.171641+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12804096,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10477","reported_seq":10685,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.755528+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:47.755528+0000","last_peered":"2024-09-10T20:28:47.755528+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:47.755528+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13046944,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10628","reported_seq":10801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.754990+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:47.754990+0000","last_peered":"2024-09-10T20:28:47.754990+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:47.754990+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13636768,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10423","reported_seq":14400,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:48.978509+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:48.978509+0000","last_peered":"2024-09-10T20:28:48.978509+0000","last_clean":"2024-09-10T20:28:48.978509+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:48.978509+0000","last_undegraded":"2024-09-10T20:28:48.978509+0000","last_fullsized":"2024-09-10T20:28:48.978509+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925520,"num_objects":1578,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10435","reported_seq":14410,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:45.662842+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:45.662842+0000","last_peered":"2024-09-10T20:28:45.662842+0000","last_clean":"2024-09-10T20:28:45.662842+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:45.662842+0000","last_undegraded":"2024-09-10T20:28:45.662842+0000","last_fullsized":"2024-09-10T20:28:45.662842+0000","mapping_epoch":39,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13983744,"num_objects":1707,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10492","reported_seq":10770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.216814+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:49.216814+0000","last_peered":"2024-09-10T20:28:49.216814+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:49.216814+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13274864,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10510","reported_seq":10813,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.132161+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:49.132161+0000","last_peered":"2024-09-10T20:28:49.132161+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:49.132161+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13556304,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10509,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10183","reported_seq":10371,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.634612+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:45.634612+0000","last_peered":"2024-09-10T20:28:45.634612+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:45.634612+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13498960,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10182,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-255}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10502","reported_seq":12973,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.248050+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:49.248050+0000","last_peered":"2024-09-10T20:28:49.248050+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:49.248050+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13038752,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10500,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10764","reported_seq":10973,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:45.668301+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:45.668301+0000","last_peered":"2024-09-10T20:28:45.668301+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:45.668301+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13800608,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10478","reported_seq":14576,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.335515+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:47.335515+0000","last_peered":"2024-09-10T20:28:47.335515+0000","last_clean":"2024-09-10T20:28:47.335515+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:47.335515+0000","last_undegraded":"2024-09-10T20:28:47.335515+0000","last_fullsized":"2024-09-10T20:28:47.335515+0000","mapping_epoch":41,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13475840,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214220648,"num_objects":26154,"num_object_clones":0,"num_object_copies":104616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26154,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139064,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428949504,"data_stored":428949504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727148,"kb_used_data":104016,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93644692,"statfs":{"total":96636764160,"available":95892164608,"internally_reserved":0,"allocated":106512384,"data_stored":106289868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733908,"kb_used_data":107140,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93637932,"statfs":{"total":96636764160,"available":95885242368,"internally_reserved":0,"allocated":109711360,"data_stored":109488844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584508,"kb_used_data":105772,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93787332,"statfs":{"total":96636764160,"available":96038227968,"internally_reserved":0,"allocated":108310528,"data_stored":108088012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732440,"kb_used_data":105204,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93639400,"statfs":{"total":96636764160,"available":95886745600,"internally_reserved":0,"allocated":107728896,"data_stored":107506380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106942464,"data_stored":106942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107388928,"data_stored":107388928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":108818432,"data_stored":108818432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105799680,"data_stored":105799680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:53.394 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:53.723 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:53.737 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-10T20:28:52.085386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":211935080,"num_objects":25875,"num_object_clones":0,"num_object_copies":103500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139343,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160594,"ondisk_log_size":160594,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778168,"kb_used_data":418136,"kb_used_omap":14,"kb_used_meta":2359921,"kb_avail":374709192,"statfs":{"total":386547056640,"available":383702212608,"internally_reserved":0,"allocated":428171264,"data_stored":427281200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2416559670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,2,7,22,15,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19316736,"num_objects":-2358,"num_object_clones":0,"num_object_copies":-9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004813"},"pg_stats":[{"pgid":"1.f","version":"42'10548","reported_seq":10835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.217832+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:49.217832+0000","last_peered":"2024-09-10T20:28:49.217832+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:49.217832+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12894208,"num_objects":1574,"num_object_clones":0,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10605","reported_seq":10742,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.270427+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:47.270427+0000","last_peered":"2024-09-10T20:28:47.270427+0000","last_clean":"2024-09-10T20:28:47.270427+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:47.270427+0000","last_undegraded":"2024-09-10T20:28:47.270427+0000","last_fullsized":"2024-09-10T20:28:47.270427+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13492224,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10540","reported_seq":10748,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.514726+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:50.514726+0000","last_peered":"2024-09-10T20:28:50.514726+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:50.514726+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13105768,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10539,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10719","reported_seq":14892,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.349465+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:47.349465+0000","last_peered":"2024-09-10T20:28:47.349465+0000","last_clean":"2024-09-10T20:28:47.349465+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:47.349465+0000","last_undegraded":"2024-09-10T20:28:47.349465+0000","last_fullsized":"2024-09-10T20:28:47.349465+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10541","reported_seq":10866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.027119+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:49.027119+0000","last_peered":"2024-09-10T20:28:49.027119+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:49.027119+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13333664,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10535","reported_seq":10762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.171641+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:49.171641+0000","last_peered":"2024-09-10T20:28:49.171641+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:49.171641+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12804096,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10477","reported_seq":10685,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.755528+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:47.755528+0000","last_peered":"2024-09-10T20:28:47.755528+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:47.755528+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13046944,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10628","reported_seq":10801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:47.754990+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:47.754990+0000","last_peered":"2024-09-10T20:28:47.754990+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:47.754990+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13636768,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10423","reported_seq":14400,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:48.978509+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:48.978509+0000","last_peered":"2024-09-10T20:28:48.978509+0000","last_clean":"2024-09-10T20:28:48.978509+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:48.978509+0000","last_undegraded":"2024-09-10T20:28:48.978509+0000","last_fullsized":"2024-09-10T20:28:48.978509+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925520,"num_objects":1578,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10507","reported_seq":14482,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:50.758462+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:50.758462+0000","last_peered":"2024-09-10T20:28:50.758462+0000","last_clean":"2024-09-10T20:28:50.758462+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:50.758462+0000","last_undegraded":"2024-09-10T20:28:50.758462+0000","last_fullsized":"2024-09-10T20:28:50.758462+0000","mapping_epoch":39,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13393920,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3145,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10492","reported_seq":10770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.216814+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:49.216814+0000","last_peered":"2024-09-10T20:28:49.216814+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:49.216814+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13274864,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10510","reported_seq":10813,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.132161+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:49.132161+0000","last_peered":"2024-09-10T20:28:49.132161+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:49.132161+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13556304,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10509,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10255","reported_seq":10443,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.693865+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:50.693865+0000","last_peered":"2024-09-10T20:28:50.693865+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:50.693865+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12909136,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10254,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10502","reported_seq":12973,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.248050+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:49.248050+0000","last_peered":"2024-09-10T20:28:49.248050+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:49.248050+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13038752,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10500,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10834","reported_seq":11044,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.667100+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:50.667100+0000","last_peered":"2024-09-10T20:28:50.667100+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:50.667100+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227168,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10478","reported_seq":14576,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:47.335515+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:47.335515+0000","last_peered":"2024-09-10T20:28:47.335515+0000","last_clean":"2024-09-10T20:28:47.335515+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:47.335515+0000","last_undegraded":"2024-09-10T20:28:47.335515+0000","last_fullsized":"2024-09-10T20:28:47.335515+0000","mapping_epoch":41,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13475840,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":211935080,"num_objects":25875,"num_object_clones":0,"num_object_copies":103500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139343,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":424730624,"data_stored":424730624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160594,"ondisk_log_size":160594,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727148,"kb_used_data":104016,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93644692,"statfs":{"total":96636764160,"available":95892164608,"internally_reserved":0,"allocated":106512384,"data_stored":106289868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734072,"kb_used_data":103144,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93637768,"statfs":{"total":96636764160,"available":95885074432,"internally_reserved":0,"allocated":105619456,"data_stored":105396940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584508,"kb_used_data":105772,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93787332,"statfs":{"total":96636764160,"available":96038227968,"internally_reserved":0,"allocated":108310528,"data_stored":108088012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732440,"kb_used_data":105204,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93639400,"statfs":{"total":96636764160,"available":95886745600,"internally_reserved":0,"allocated":107728896,"data_stored":107506380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106942464,"data_stored":106942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107388928,"data_stored":107388928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104599552,"data_stored":104599552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105799680,"data_stored":105799680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:53.738 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:54.248 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:54.248 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:54.260 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-10T20:28:54.085850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209370984,"num_objects":25562,"num_object_clones":0,"num_object_copies":102248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139656,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778292,"kb_used_data":409944,"kb_used_omap":14,"kb_used_meta":2368241,"kb_avail":374709068,"statfs":{"total":386547056640,"available":383702085632,"internally_reserved":0,"allocated":419782656,"data_stored":418892592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2425079350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,20,15,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19644416,"num_objects":-2398,"num_object_clones":0,"num_object_copies":-9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2398,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004476"},"pg_stats":[{"pgid":"1.f","version":"42'10548","reported_seq":10835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.217832+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:49.217832+0000","last_peered":"2024-09-10T20:28:49.217832+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:49.217832+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12894208,"num_objects":1574,"num_object_clones":0,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10672","reported_seq":10809,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.427596+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:52.427596+0000","last_peered":"2024-09-10T20:28:52.427596+0000","last_clean":"2024-09-10T20:28:52.427596+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:52.427596+0000","last_undegraded":"2024-09-10T20:28:52.427596+0000","last_fullsized":"2024-09-10T20:28:52.427596+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12943360,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7113,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10540","reported_seq":10748,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.514726+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:50.514726+0000","last_peered":"2024-09-10T20:28:50.514726+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:50.514726+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13105768,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10539,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10783","reported_seq":14957,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.081445+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:52.081445+0000","last_peered":"2024-09-10T20:28:52.081445+0000","last_clean":"2024-09-10T20:28:52.081445+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:52.081445+0000","last_undegraded":"2024-09-10T20:28:52.081445+0000","last_fullsized":"2024-09-10T20:28:52.081445+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10541","reported_seq":10866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.027119+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:49.027119+0000","last_peered":"2024-09-10T20:28:49.027119+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:49.027119+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13333664,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10535","reported_seq":10762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.171641+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:49.171641+0000","last_peered":"2024-09-10T20:28:49.171641+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:49.171641+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12804096,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10537","reported_seq":10745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.425550+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:52.425550+0000","last_peered":"2024-09-10T20:28:52.425550+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:52.425550+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12555424,"num_objects":1533,"num_object_clones":0,"num_object_copies":6132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10688","reported_seq":10861,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.830090+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:52.830090+0000","last_peered":"2024-09-10T20:28:52.830090+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:52.830090+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13145248,"num_objects":1605,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10686,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10423","reported_seq":14400,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:48.978509+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:48.978509+0000","last_peered":"2024-09-10T20:28:48.978509+0000","last_clean":"2024-09-10T20:28:48.978509+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:48.978509+0000","last_undegraded":"2024-09-10T20:28:48.978509+0000","last_fullsized":"2024-09-10T20:28:48.978509+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925520,"num_objects":1578,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10507","reported_seq":14482,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:50.758462+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:50.758462+0000","last_peered":"2024-09-10T20:28:50.758462+0000","last_clean":"2024-09-10T20:28:50.758462+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:50.758462+0000","last_undegraded":"2024-09-10T20:28:50.758462+0000","last_fullsized":"2024-09-10T20:28:50.758462+0000","mapping_epoch":39,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13393920,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3145,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10492","reported_seq":10770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.216814+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:49.216814+0000","last_peered":"2024-09-10T20:28:49.216814+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:49.216814+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13274864,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10510","reported_seq":10813,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.132161+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:49.132161+0000","last_peered":"2024-09-10T20:28:49.132161+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:49.132161+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13556304,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10509,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10255","reported_seq":10443,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.693865+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:50.693865+0000","last_peered":"2024-09-10T20:28:50.693865+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:50.693865+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12909136,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10254,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10502","reported_seq":12973,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.248050+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:49.248050+0000","last_peered":"2024-09-10T20:28:49.248050+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:49.248050+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13038752,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10500,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10834","reported_seq":11044,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.667100+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:50.667100+0000","last_peered":"2024-09-10T20:28:50.667100+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:50.667100+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227168,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10540","reported_seq":14640,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.200027+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:52.200027+0000","last_peered":"2024-09-10T20:28:52.200027+0000","last_clean":"2024-09-10T20:28:52.200027+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:52.200027+0000","last_undegraded":"2024-09-10T20:28:52.200027+0000","last_fullsized":"2024-09-10T20:28:52.200027+0000","mapping_epoch":41,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12967936,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1413,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":209370984,"num_objects":25562,"num_object_clones":0,"num_object_copies":102248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139656,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":416464896,"data_stored":416464896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727148,"kb_used_data":104016,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93644692,"statfs":{"total":96636764160,"available":95892164608,"internally_reserved":0,"allocated":106512384,"data_stored":106289868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734072,"kb_used_data":103144,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93637768,"statfs":{"total":96636764160,"available":95885074432,"internally_reserved":0,"allocated":105619456,"data_stored":105396940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584488,"kb_used_data":101584,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93787352,"statfs":{"total":96636764160,"available":96038248448,"internally_reserved":0,"allocated":104022016,"data_stored":103799500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494464722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732584,"kb_used_data":101200,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93639256,"statfs":{"total":96636764160,"available":95886598144,"internally_reserved":0,"allocated":103628800,"data_stored":103406284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102825984,"data_stored":102825984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103239680,"data_stored":103239680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104599552,"data_stored":104599552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105799680,"data_stored":105799680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:54.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:54.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10548", 'reported_seq': 10835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:49.217832+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:49.217832+0000', 'last_peered': '2024-09-10T20:28:49.217832+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:49.217832+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12894208, 'num_objects': 1574, 'num_object_clones': 0, 'num_object_copies': 6296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10548, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -281}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10540", 'reported_seq': 10748, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:50.514726+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:50.514726+0000', 'last_peered': '2024-09-10T20:28:50.514726+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:50.514726+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13105768, 'num_objects': 1601, 'num_object_clones': 0, 'num_object_copies': 6404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1601, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10539, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -307}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10541", 'reported_seq': 10866, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:49.027119+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:49.027119+0000', 'last_peered': '2024-09-10T20:28:49.027119+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:49.027119+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13333664, 'num_objects': 1628, 'num_object_clones': 0, 'num_object_copies': 6512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10539, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -227}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10535", 'reported_seq': 10762, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:49.171641+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:49.171641+0000', 'last_peered': '2024-09-10T20:28:49.171641+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:49.171641+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12804096, 'num_objects': 1563, 'num_object_clones': 0, 'num_object_copies': 6252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10535, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10537", 'reported_seq': 10745, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:52.425550+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:52.425550+0000', 'last_peered': '2024-09-10T20:28:52.425550+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:52.425550+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12555424, 'num_objects': 1533, 'num_object_clones': 0, 'num_object_copies': 6132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10535, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -360}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:54.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10688", 'reported_seq': 10861, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:52.830090+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:52.830090+0000', 'last_peered': '2024-09-10T20:28:52.830090+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:52.830090+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13145248, 'num_objects': 1605, 'num_object_clones': 0, 'num_object_copies': 6420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10686, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10492", 'reported_seq': 10770, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:49.216814+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:49.216814+0000', 'last_peered': '2024-09-10T20:28:49.216814+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:49.216814+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13274864, 'num_objects': 1621, 'num_object_clones': 0, 'num_object_copies': 6484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10489, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -235}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10510", 'reported_seq': 10813, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:49.132161+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:49.132161+0000', 'last_peered': '2024-09-10T20:28:49.132161+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:49.132161+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13556304, 'num_objects': 1655, 'num_object_clones': 0, 'num_object_copies': 6620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10509, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -205}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10255", 'reported_seq': 10443, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:50.693865+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:50.693865+0000', 'last_peered': '2024-09-10T20:28:50.693865+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:50.693865+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12909136, 'num_objects': 1576, 'num_object_clones': 0, 'num_object_copies': 6304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10254, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10502", 'reported_seq': 12973, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:49.248050+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:49.248050+0000', 'last_peered': '2024-09-10T20:28:49.248050+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:49.248050+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13038752, 'num_objects': 1592, 'num_object_clones': 0, 'num_object_copies': 6368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 779, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10500, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1200, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:54.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10834", 'reported_seq': 11044, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:50.667100+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:50.667100+0000', 'last_peered': '2024-09-10T20:28:50.667100+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:50.667100+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13227168, 'num_objects': 1615, 'num_object_clones': 0, 'num_object_copies': 6460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10832, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:54.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:28:54.902 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:54.903 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19688/102248 objects degraded (19.255%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":623,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25562,"data_bytes":209370984,"bytes_used":2844971008,"bytes_avail":383702085632,"bytes_total":386547056640,"degraded_objects":19688,"degraded_total":102248,"degraded_ratio":0.19255144354901804,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:28:54.938 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:28:54.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:55.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:55.578 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-10T20:28:54.085850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209370984,"num_objects":25562,"num_object_clones":0,"num_object_copies":102248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139656,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778292,"kb_used_data":409944,"kb_used_omap":14,"kb_used_meta":2368241,"kb_avail":374709068,"statfs":{"total":386547056640,"available":383702085632,"internally_reserved":0,"allocated":419782656,"data_stored":418892592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2425079350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,20,15,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19644416,"num_objects":-2398,"num_object_clones":0,"num_object_copies":-9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2398,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004476"},"pg_stats":[{"pgid":"1.f","version":"42'10548","reported_seq":10835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.217832+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:49.217832+0000","last_peered":"2024-09-10T20:28:49.217832+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:49.217832+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12894208,"num_objects":1574,"num_object_clones":0,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10672","reported_seq":10809,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.427596+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:52.427596+0000","last_peered":"2024-09-10T20:28:52.427596+0000","last_clean":"2024-09-10T20:28:52.427596+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:52.427596+0000","last_undegraded":"2024-09-10T20:28:52.427596+0000","last_fullsized":"2024-09-10T20:28:52.427596+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12943360,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7113,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10540","reported_seq":10748,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.514726+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:50.514726+0000","last_peered":"2024-09-10T20:28:50.514726+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:50.514726+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13105768,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10539,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10783","reported_seq":14957,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.081445+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:52.081445+0000","last_peered":"2024-09-10T20:28:52.081445+0000","last_clean":"2024-09-10T20:28:52.081445+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:52.081445+0000","last_undegraded":"2024-09-10T20:28:52.081445+0000","last_fullsized":"2024-09-10T20:28:52.081445+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10541","reported_seq":10866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.027119+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:49.027119+0000","last_peered":"2024-09-10T20:28:49.027119+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:49.027119+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13333664,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10535","reported_seq":10762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.171641+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:49.171641+0000","last_peered":"2024-09-10T20:28:49.171641+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:49.171641+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12804096,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10537","reported_seq":10745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.425550+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:52.425550+0000","last_peered":"2024-09-10T20:28:52.425550+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:52.425550+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12555424,"num_objects":1533,"num_object_clones":0,"num_object_copies":6132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10688","reported_seq":10861,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.830090+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:52.830090+0000","last_peered":"2024-09-10T20:28:52.830090+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:52.830090+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13145248,"num_objects":1605,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10686,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10423","reported_seq":14400,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:48.978509+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:48.978509+0000","last_peered":"2024-09-10T20:28:48.978509+0000","last_clean":"2024-09-10T20:28:48.978509+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:48.978509+0000","last_undegraded":"2024-09-10T20:28:48.978509+0000","last_fullsized":"2024-09-10T20:28:48.978509+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925520,"num_objects":1578,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10507","reported_seq":14482,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:50.758462+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:50.758462+0000","last_peered":"2024-09-10T20:28:50.758462+0000","last_clean":"2024-09-10T20:28:50.758462+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:50.758462+0000","last_undegraded":"2024-09-10T20:28:50.758462+0000","last_fullsized":"2024-09-10T20:28:50.758462+0000","mapping_epoch":39,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13393920,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3145,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10492","reported_seq":10770,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.216814+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:49.216814+0000","last_peered":"2024-09-10T20:28:49.216814+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:49.216814+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13274864,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10510","reported_seq":10813,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.132161+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:49.132161+0000","last_peered":"2024-09-10T20:28:49.132161+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:49.132161+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13556304,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10509,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-205}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10255","reported_seq":10443,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.693865+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:50.693865+0000","last_peered":"2024-09-10T20:28:50.693865+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:50.693865+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12909136,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10254,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10502","reported_seq":12973,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:49.248050+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:49.248050+0000","last_peered":"2024-09-10T20:28:49.248050+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:49.248050+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13038752,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10500,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10834","reported_seq":11044,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:50.667100+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:50.667100+0000","last_peered":"2024-09-10T20:28:50.667100+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:50.667100+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227168,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10540","reported_seq":14640,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.200027+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:52.200027+0000","last_peered":"2024-09-10T20:28:52.200027+0000","last_clean":"2024-09-10T20:28:52.200027+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:52.200027+0000","last_undegraded":"2024-09-10T20:28:52.200027+0000","last_fullsized":"2024-09-10T20:28:52.200027+0000","mapping_epoch":41,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12967936,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1413,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":209370984,"num_objects":25562,"num_object_clones":0,"num_object_copies":102248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139656,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":416464896,"data_stored":416464896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727148,"kb_used_data":104016,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93644692,"statfs":{"total":96636764160,"available":95892164608,"internally_reserved":0,"allocated":106512384,"data_stored":106289868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734072,"kb_used_data":103144,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93637768,"statfs":{"total":96636764160,"available":95885074432,"internally_reserved":0,"allocated":105619456,"data_stored":105396940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584488,"kb_used_data":101584,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93787352,"statfs":{"total":96636764160,"available":96038248448,"internally_reserved":0,"allocated":104022016,"data_stored":103799500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494464722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732584,"kb_used_data":101200,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93639256,"statfs":{"total":96636764160,"available":95886598144,"internally_reserved":0,"allocated":103628800,"data_stored":103406284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102825984,"data_stored":102825984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103239680,"data_stored":103239680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104599552,"data_stored":104599552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105799680,"data_stored":105799680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:58.615 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:59.225 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:59.226 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:59.260 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-10T20:28:58.087218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202932072,"num_objects":24776,"num_object_clones":0,"num_object_copies":99104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140442,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160893,"ondisk_log_size":160893,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796540,"kb_used_data":397680,"kb_used_omap":14,"kb_used_meta":2398769,"kb_avail":374690820,"statfs":{"total":386547056640,"available":383683399680,"internally_reserved":0,"allocated":407224320,"data_stored":406334256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2456340022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,14,12,8,0,0,0,13,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17596416,"num_objects":-2148,"num_object_clones":0,"num_object_copies":-8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2148,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004201"},"pg_stats":[{"pgid":"1.f","version":"42'10602","reported_seq":10889,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.703785+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:52.703785+0000","last_peered":"2024-09-10T20:28:52.703785+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:52.703785+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12451840,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10602,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10706","reported_seq":10843,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.502513+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:57.502513+0000","last_peered":"2024-09-10T20:28:57.502513+0000","last_clean":"2024-09-10T20:28:57.502513+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:57.502513+0000","last_undegraded":"2024-09-10T20:28:57.502513+0000","last_fullsized":"2024-09-10T20:28:57.502513+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12664832,"num_objects":1546,"num_object_clones":0,"num_object_copies":6184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10586","reported_seq":10793,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.825067+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:55.825067+0000","last_peered":"2024-09-10T20:28:55.825067+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:55.825067+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12728936,"num_objects":1555,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10585,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10832","reported_seq":15006,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.333905+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:57.333905+0000","last_peered":"2024-09-10T20:28:57.333905+0000","last_clean":"2024-09-10T20:28:57.333905+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:57.333905+0000","last_undegraded":"2024-09-10T20:28:57.333905+0000","last_fullsized":"2024-09-10T20:28:57.333905+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12894208,"num_objects":1574,"num_object_clones":0,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10592","reported_seq":10917,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.960089+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:52.960089+0000","last_peered":"2024-09-10T20:28:52.960089+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:52.960089+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12915872,"num_objects":1577,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10591","reported_seq":10817,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.892027+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:52.892027+0000","last_peered":"2024-09-10T20:28:52.892027+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:52.892027+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12345344,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10597","reported_seq":10805,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:57.867940+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:57.867940+0000","last_peered":"2024-09-10T20:28:57.867940+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:57.867940+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12063904,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10737","reported_seq":10910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:57.946138+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:57.946138+0000","last_peered":"2024-09-10T20:28:57.946138+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:57.946138+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12743840,"num_objects":1556,"num_object_clones":0,"num_object_copies":6224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10473","reported_seq":14450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.944577+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:52.944577+0000","last_peered":"2024-09-10T20:28:52.944577+0000","last_clean":"2024-09-10T20:28:52.944577+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:52.944577+0000","last_undegraded":"2024-09-10T20:28:52.944577+0000","last_fullsized":"2024-09-10T20:28:52.944577+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12515920,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7026,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10555","reported_seq":14530,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:55.686635+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:55.686635+0000","last_peered":"2024-09-10T20:28:55.686635+0000","last_clean":"2024-09-10T20:28:55.686635+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:55.686635+0000","last_undegraded":"2024-09-10T20:28:55.686635+0000","last_fullsized":"2024-09-10T20:28:55.686635+0000","mapping_epoch":39,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10524","reported_seq":10802,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.633256+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:52.633256+0000","last_peered":"2024-09-10T20:28:52.633256+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:52.633256+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13012720,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10555","reported_seq":10858,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.897755+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:52.897755+0000","last_peered":"2024-09-10T20:28:52.897755+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:52.897755+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13187664,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10318","reported_seq":10505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.795998+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:55.795998+0000","last_peered":"2024-09-10T20:28:55.795998+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:55.795998+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12393040,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-390}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10549","reported_seq":13675,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:54.286573+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:54.286573+0000","last_peered":"2024-09-10T20:28:54.286573+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:54.286573+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12653728,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10891","reported_seq":11097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.745906+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:55.745906+0000","last_peered":"2024-09-10T20:28:55.745906+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:55.745906+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12760224,"num_objects":1558,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10889,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10585","reported_seq":14685,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.415837+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:57.415837+0000","last_peered":"2024-09-10T20:28:57.415837+0000","last_clean":"2024-09-10T20:28:57.415837+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:57.415837+0000","last_undegraded":"2024-09-10T20:28:57.415837+0000","last_fullsized":"2024-09-10T20:28:57.415837+0000","mapping_epoch":41,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12599296,"num_objects":1538,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":202932072,"num_objects":24776,"num_object_clones":0,"num_object_copies":99104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140442,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403939328,"data_stored":403939328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160893,"ondisk_log_size":160893,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117110,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727936,"kb_used_data":100656,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93643904,"statfs":{"total":96636764160,"available":95891357696,"internally_reserved":0,"allocated":103071744,"data_stored":102849228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,13,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731168,"kb_used_data":100240,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93640672,"statfs":{"total":96636764160,"available":95888048128,"internally_reserved":0,"allocated":102645760,"data_stored":102423244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608052,"kb_used_data":98776,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93763788,"statfs":{"total":96636764160,"available":96014118912,"internally_reserved":0,"allocated":101146624,"data_stored":100924108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729384,"kb_used_data":98008,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93642456,"statfs":{"total":96636764160,"available":95889874944,"internally_reserved":0,"allocated":100360192,"data_stored":100137676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99614720,"data_stored":99614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":100061184,"data_stored":100061184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101543936,"data_stored":101543936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102719488,"data_stored":102719488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:59.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:28:59.852 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:28:59.852 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:28:59.873 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-10T20:28:58.087218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202932072,"num_objects":24776,"num_object_clones":0,"num_object_copies":99104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140442,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160893,"ondisk_log_size":160893,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796540,"kb_used_data":397680,"kb_used_omap":14,"kb_used_meta":2398769,"kb_avail":374690820,"statfs":{"total":386547056640,"available":383683399680,"internally_reserved":0,"allocated":407224320,"data_stored":406334256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2456340022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,14,12,8,0,0,0,13,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17596416,"num_objects":-2148,"num_object_clones":0,"num_object_copies":-8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2148,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004201"},"pg_stats":[{"pgid":"1.f","version":"42'10602","reported_seq":10889,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.703785+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:52.703785+0000","last_peered":"2024-09-10T20:28:52.703785+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:52.703785+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12451840,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10602,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10706","reported_seq":10843,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.502513+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:57.502513+0000","last_peered":"2024-09-10T20:28:57.502513+0000","last_clean":"2024-09-10T20:28:57.502513+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:57.502513+0000","last_undegraded":"2024-09-10T20:28:57.502513+0000","last_fullsized":"2024-09-10T20:28:57.502513+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12664832,"num_objects":1546,"num_object_clones":0,"num_object_copies":6184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10586","reported_seq":10793,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.825067+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:55.825067+0000","last_peered":"2024-09-10T20:28:55.825067+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:55.825067+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12728936,"num_objects":1555,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10585,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10832","reported_seq":15006,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.333905+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:57.333905+0000","last_peered":"2024-09-10T20:28:57.333905+0000","last_clean":"2024-09-10T20:28:57.333905+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:57.333905+0000","last_undegraded":"2024-09-10T20:28:57.333905+0000","last_fullsized":"2024-09-10T20:28:57.333905+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12894208,"num_objects":1574,"num_object_clones":0,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10592","reported_seq":10917,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.960089+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:52.960089+0000","last_peered":"2024-09-10T20:28:52.960089+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:52.960089+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12915872,"num_objects":1577,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10591","reported_seq":10817,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.892027+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:52.892027+0000","last_peered":"2024-09-10T20:28:52.892027+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:52.892027+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12345344,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10597","reported_seq":10805,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:57.867940+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:57.867940+0000","last_peered":"2024-09-10T20:28:57.867940+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:57.867940+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12063904,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10737","reported_seq":10910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:57.946138+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:57.946138+0000","last_peered":"2024-09-10T20:28:57.946138+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:57.946138+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12743840,"num_objects":1556,"num_object_clones":0,"num_object_copies":6224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10473","reported_seq":14450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:52.944577+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:52.944577+0000","last_peered":"2024-09-10T20:28:52.944577+0000","last_clean":"2024-09-10T20:28:52.944577+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:52.944577+0000","last_undegraded":"2024-09-10T20:28:52.944577+0000","last_fullsized":"2024-09-10T20:28:52.944577+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12515920,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7026,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10555","reported_seq":14530,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:55.686635+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:55.686635+0000","last_peered":"2024-09-10T20:28:55.686635+0000","last_clean":"2024-09-10T20:28:55.686635+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:55.686635+0000","last_undegraded":"2024-09-10T20:28:55.686635+0000","last_fullsized":"2024-09-10T20:28:55.686635+0000","mapping_epoch":39,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10524","reported_seq":10802,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.633256+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:52.633256+0000","last_peered":"2024-09-10T20:28:52.633256+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:52.633256+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13012720,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10555","reported_seq":10858,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:52.897755+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:52.897755+0000","last_peered":"2024-09-10T20:28:52.897755+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:52.897755+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13187664,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10318","reported_seq":10505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.795998+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:55.795998+0000","last_peered":"2024-09-10T20:28:55.795998+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:55.795998+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12393040,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-390}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10549","reported_seq":13675,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:54.286573+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:54.286573+0000","last_peered":"2024-09-10T20:28:54.286573+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:54.286573+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12653728,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10891","reported_seq":11097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.745906+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:55.745906+0000","last_peered":"2024-09-10T20:28:55.745906+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:55.745906+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12760224,"num_objects":1558,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10889,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10585","reported_seq":14685,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.415837+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:57.415837+0000","last_peered":"2024-09-10T20:28:57.415837+0000","last_clean":"2024-09-10T20:28:57.415837+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:57.415837+0000","last_undegraded":"2024-09-10T20:28:57.415837+0000","last_fullsized":"2024-09-10T20:28:57.415837+0000","mapping_epoch":41,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12599296,"num_objects":1538,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":202932072,"num_objects":24776,"num_object_clones":0,"num_object_copies":99104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140442,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403939328,"data_stored":403939328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160893,"ondisk_log_size":160893,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117110,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727936,"kb_used_data":100656,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93643904,"statfs":{"total":96636764160,"available":95891357696,"internally_reserved":0,"allocated":103071744,"data_stored":102849228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,13,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731168,"kb_used_data":100240,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93640672,"statfs":{"total":96636764160,"available":95888048128,"internally_reserved":0,"allocated":102645760,"data_stored":102423244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608052,"kb_used_data":98776,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93763788,"statfs":{"total":96636764160,"available":96014118912,"internally_reserved":0,"allocated":101146624,"data_stored":100924108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729384,"kb_used_data":98008,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93642456,"statfs":{"total":96636764160,"available":95889874944,"internally_reserved":0,"allocated":100360192,"data_stored":100137676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99614720,"data_stored":99614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":100061184,"data_stored":100061184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101543936,"data_stored":101543936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102719488,"data_stored":102719488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:28:59.874 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:28:59.874 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10602", 'reported_seq': 10889, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:52.703785+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:28:52.703785+0000', 'last_peered': '2024-09-10T20:28:52.703785+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:28:52.703785+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12451840, 'num_objects': 1520, 'num_object_clones': 0, 'num_object_copies': 6080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10602, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10586", 'reported_seq': 10793, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:55.825067+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:28:55.825067+0000', 'last_peered': '2024-09-10T20:28:55.825067+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:28:55.825067+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12728936, 'num_objects': 1555, 'num_object_clones': 0, 'num_object_copies': 6220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1555, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10585, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -353}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10592", 'reported_seq': 10917, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:52.960089+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:28:52.960089+0000', 'last_peered': '2024-09-10T20:28:52.960089+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:28:52.960089+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12915872, 'num_objects': 1577, 'num_object_clones': 0, 'num_object_copies': 6308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10590, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10591", 'reported_seq': 10817, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:52.892027+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:28:52.892027+0000', 'last_peered': '2024-09-10T20:28:52.892027+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:28:52.892027+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12345344, 'num_objects': 1507, 'num_object_clones': 0, 'num_object_copies': 6028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10591, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -350}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10597", 'reported_seq': 10805, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:57.867940+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:28:57.867940+0000', 'last_peered': '2024-09-10T20:28:57.867940+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:28:57.867940+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12063904, 'num_objects': 1473, 'num_object_clones': 0, 'num_object_copies': 5892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10595, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -420}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:59.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:28:59.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10737", 'reported_seq': 10910, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:57.946138+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:28:57.946138+0000', 'last_peered': '2024-09-10T20:28:57.946138+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:28:57.946138+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12743840, 'num_objects': 1556, 'num_object_clones': 0, 'num_object_copies': 6224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10735, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:28:59.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:28:59.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10524", 'reported_seq': 10802, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:52.633256+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:28:52.633256+0000', 'last_peered': '2024-09-10T20:28:52.633256+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:28:52.633256+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13012720, 'num_objects': 1589, 'num_object_clones': 0, 'num_object_copies': 6356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10521, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -267}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:59.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:28:59.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10555", 'reported_seq': 10858, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:52.897755+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:28:52.897755+0000', 'last_peered': '2024-09-10T20:28:52.897755+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:28:52.897755+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13187664, 'num_objects': 1610, 'num_object_clones': 0, 'num_object_copies': 6440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10554, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:59.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:28:59.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10318", 'reported_seq': 10505, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:55.795998+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:28:55.795998+0000', 'last_peered': '2024-09-10T20:28:55.795998+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:28:55.795998+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12393040, 'num_objects': 1513, 'num_object_clones': 0, 'num_object_copies': 6052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10317, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -390}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:59.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:28:59.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10549", 'reported_seq': 13675, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:54.286573+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:28:54.286573+0000', 'last_peered': '2024-09-10T20:28:54.286573+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:28:54.286573+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12653728, 'num_objects': 1545, 'num_object_clones': 0, 'num_object_copies': 6180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 451, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10547, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1528, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1094}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:28:59.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:28:59.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'10891", 'reported_seq': 11097, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:28:55.745906+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:28:55.745906+0000', 'last_peered': '2024-09-10T20:28:55.745906+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:28:55.745906+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12760224, 'num_objects': 1558, 'num_object_clones': 0, 'num_object_copies': 6232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10889, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:28:59.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:00.648 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:00.648 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19360/99104 objects degraded (19.535%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":629,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24776,"data_bytes":202932072,"bytes_used":2863656960,"bytes_avail":383683399680,"bytes_total":386547056640,"degraded_objects":19360,"degraded_total":99104,"degraded_ratio":0.19535033903777849,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":404,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:00.681 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:00.681 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:01.253 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:01.282 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-10T20:29:00.087879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199327592,"num_objects":24336,"num_object_clones":0,"num_object_copies":97344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24336,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140882,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796932,"kb_used_data":393904,"kb_used_omap":14,"kb_used_meta":2402929,"kb_avail":374690428,"statfs":{"total":386547056640,"available":383682998272,"internally_reserved":0,"allocated":403357696,"data_stored":402467632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2460599862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,17,12,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003984"},"pg_stats":[{"pgid":"1.f","version":"42'10665","reported_seq":10952,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.159177+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:59.159177+0000","last_peered":"2024-09-10T20:28:59.159177+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:59.159177+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11935744,"num_objects":1457,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10706","reported_seq":10843,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.502513+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:28:57.502513+0000","last_peered":"2024-09-10T20:28:57.502513+0000","last_clean":"2024-09-10T20:28:57.502513+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:28:57.502513+0000","last_undegraded":"2024-09-10T20:28:57.502513+0000","last_fullsized":"2024-09-10T20:28:57.502513+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12664832,"num_objects":1546,"num_object_clones":0,"num_object_copies":6184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10586","reported_seq":10793,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.825067+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:28:55.825067+0000","last_peered":"2024-09-10T20:28:55.825067+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:28:55.825067+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12728936,"num_objects":1555,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10585,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10832","reported_seq":15006,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.333905+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:28:57.333905+0000","last_peered":"2024-09-10T20:28:57.333905+0000","last_clean":"2024-09-10T20:28:57.333905+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:28:57.333905+0000","last_undegraded":"2024-09-10T20:28:57.333905+0000","last_fullsized":"2024-09-10T20:28:57.333905+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12894208,"num_objects":1574,"num_object_clones":0,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10655","reported_seq":10978,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.377030+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:59.377030+0000","last_peered":"2024-09-10T20:28:59.377030+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:59.377030+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12399776,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10645","reported_seq":10872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.228048+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:59.228048+0000","last_peered":"2024-09-10T20:28:59.228048+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:59.228048+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11902976,"num_objects":1453,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10597","reported_seq":10805,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:57.867940+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:28:57.867940+0000","last_peered":"2024-09-10T20:28:57.867940+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:28:57.867940+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12063904,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10737","reported_seq":10910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:57.946138+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:28:57.946138+0000","last_peered":"2024-09-10T20:28:57.946138+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:28:57.946138+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12743840,"num_objects":1556,"num_object_clones":0,"num_object_copies":6224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10537","reported_seq":14514,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:59.119107+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:59.119107+0000","last_peered":"2024-09-10T20:28:59.119107+0000","last_clean":"2024-09-10T20:28:59.119107+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:59.119107+0000","last_undegraded":"2024-09-10T20:28:59.119107+0000","last_fullsized":"2024-09-10T20:28:59.119107+0000","mapping_epoch":33,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11991632,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10555","reported_seq":14530,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:55.686635+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:28:55.686635+0000","last_peered":"2024-09-10T20:28:55.686635+0000","last_clean":"2024-09-10T20:28:55.686635+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:28:55.686635+0000","last_undegraded":"2024-09-10T20:28:55.686635+0000","last_fullsized":"2024-09-10T20:28:55.686635+0000","mapping_epoch":39,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10594","reported_seq":10872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.101305+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:59.101305+0000","last_peered":"2024-09-10T20:28:59.101305+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:59.101305+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12439280,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10626","reported_seq":10929,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.265314+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:59.265314+0000","last_peered":"2024-09-10T20:28:59.265314+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:59.265314+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12606032,"num_objects":1539,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-321}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10318","reported_seq":10505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.795998+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:28:55.795998+0000","last_peered":"2024-09-10T20:28:55.795998+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:28:55.795998+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12393040,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-390}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10604","reported_seq":13857,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.318463+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:59.318463+0000","last_peered":"2024-09-10T20:28:59.318463+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:59.318463+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203168,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10602,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1591,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1102}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10891","reported_seq":11097,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:55.745906+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:28:55.745906+0000","last_peered":"2024-09-10T20:28:55.745906+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:28:55.745906+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12760224,"num_objects":1558,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10889,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10585","reported_seq":14685,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:57.415837+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:28:57.415837+0000","last_peered":"2024-09-10T20:28:57.415837+0000","last_clean":"2024-09-10T20:28:57.415837+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:28:57.415837+0000","last_undegraded":"2024-09-10T20:28:57.415837+0000","last_fullsized":"2024-09-10T20:28:57.415837+0000","mapping_epoch":41,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12599296,"num_objects":1538,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":199327592,"num_objects":24336,"num_object_clones":0,"num_object_copies":97344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24336,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140882,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399663104,"data_stored":399663104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728328,"kb_used_data":96880,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93643512,"statfs":{"total":96636764160,"available":95890956288,"internally_reserved":0,"allocated":99205120,"data_stored":98982604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731168,"kb_used_data":100240,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93640672,"statfs":{"total":96636764160,"available":95888048128,"internally_reserved":0,"allocated":102645760,"data_stored":102423244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608052,"kb_used_data":98776,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93763788,"statfs":{"total":96636764160,"available":96014118912,"internally_reserved":0,"allocated":101146624,"data_stored":100924108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729384,"kb_used_data":98008,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93642456,"statfs":{"total":96636764160,"available":95889874944,"internally_reserved":0,"allocated":100360192,"data_stored":100137676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99614720,"data_stored":99614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":100061184,"data_stored":100061184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101543936,"data_stored":101543936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98443264,"data_stored":98443264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:04.284 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:05.577 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:05.600 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-10T20:29:04.089161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194502504,"num_objects":23747,"num_object_clones":0,"num_object_copies":94988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141471,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796244,"kb_used_data":380744,"kb_used_omap":14,"kb_used_meta":2415409,"kb_avail":374691116,"statfs":{"total":386547056640,"available":383683702784,"internally_reserved":0,"allocated":389881856,"data_stored":388991792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2473379382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,13,7,19,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17432576,"num_objects":-2128,"num_object_clones":0,"num_object_copies":-8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2128,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003775"},"pg_stats":[{"pgid":"1.f","version":"42'10665","reported_seq":10952,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.159177+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:28:59.159177+0000","last_peered":"2024-09-10T20:28:59.159177+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:28:59.159177+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11935744,"num_objects":1457,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10768","reported_seq":10905,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.369112+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:02.369112+0000","last_peered":"2024-09-10T20:29:02.369112+0000","last_clean":"2024-09-10T20:29:02.369112+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:02.369112+0000","last_undegraded":"2024-09-10T20:29:02.369112+0000","last_fullsized":"2024-09-10T20:29:02.369112+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12156928,"num_objects":1484,"num_object_clones":0,"num_object_copies":5936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10658","reported_seq":10866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:00.879240+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:00.879240+0000","last_peered":"2024-09-10T20:29:00.879240+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:00.879240+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12139112,"num_objects":1483,"num_object_clones":0,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10657,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-425}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10889","reported_seq":15061,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.635891+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:02.635891+0000","last_peered":"2024-09-10T20:29:02.635891+0000","last_clean":"2024-09-10T20:29:02.635891+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:02.635891+0000","last_undegraded":"2024-09-10T20:29:02.635891+0000","last_fullsized":"2024-09-10T20:29:02.635891+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427264,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10655","reported_seq":10978,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.377030+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:28:59.377030+0000","last_peered":"2024-09-10T20:28:59.377030+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:28:59.377030+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12399776,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10645","reported_seq":10872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.228048+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:28:59.228048+0000","last_peered":"2024-09-10T20:28:59.228048+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:28:59.228048+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11902976,"num_objects":1453,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10655","reported_seq":10863,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:02.807463+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:02.807463+0000","last_peered":"2024-09-10T20:29:02.807463+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:02.807463+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11588768,"num_objects":1415,"num_object_clones":0,"num_object_copies":5660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10803","reported_seq":10976,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:02.947183+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:02.947183+0000","last_peered":"2024-09-10T20:29:02.947183+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:02.947183+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203168,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10537","reported_seq":14514,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:28:59.119107+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:28:59.119107+0000","last_peered":"2024-09-10T20:28:59.119107+0000","last_clean":"2024-09-10T20:28:59.119107+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:28:59.119107+0000","last_undegraded":"2024-09-10T20:28:59.119107+0000","last_fullsized":"2024-09-10T20:28:59.119107+0000","mapping_epoch":33,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11991632,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10634","reported_seq":14608,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:00.805706+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:00.805706+0000","last_peered":"2024-09-10T20:29:00.805706+0000","last_clean":"2024-09-10T20:29:00.805706+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:00.805706+0000","last_undegraded":"2024-09-10T20:29:00.805706+0000","last_fullsized":"2024-09-10T20:29:00.805706+0000","mapping_epoch":39,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12353536,"num_objects":1508,"num_object_clones":0,"num_object_copies":6032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3272,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10594","reported_seq":10872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.101305+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:28:59.101305+0000","last_peered":"2024-09-10T20:28:59.101305+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:28:59.101305+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12439280,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10626","reported_seq":10929,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.265314+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:28:59.265314+0000","last_peered":"2024-09-10T20:28:59.265314+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:28:59.265314+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12606032,"num_objects":1539,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-321}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10378","reported_seq":10566,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:00.956449+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:00.956449+0000","last_peered":"2024-09-10T20:29:00.956449+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:00.956449+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11901520,"num_objects":1453,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10604","reported_seq":13857,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:28:59.318463+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:28:59.318463+0000","last_peered":"2024-09-10T20:28:59.318463+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:28:59.318463+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203168,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10602,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1591,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1102}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'10962","reported_seq":11172,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:00.929732+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:00.929732+0000","last_peered":"2024-09-10T20:29:00.929732+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:00.929732+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178592,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10649","reported_seq":14747,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.562983+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:02.562983+0000","last_peered":"2024-09-10T20:29:02.562983+0000","last_clean":"2024-09-10T20:29:02.562983+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:02.562983+0000","last_undegraded":"2024-09-10T20:29:02.562983+0000","last_fullsized":"2024-09-10T20:29:02.562983+0000","mapping_epoch":41,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12075008,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194502504,"num_objects":23747,"num_object_clones":0,"num_object_copies":94988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141471,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":386662400,"data_stored":386662400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728328,"kb_used_data":96880,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93643512,"statfs":{"total":96636764160,"available":95890956288,"internally_reserved":0,"allocated":99205120,"data_stored":98982604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730460,"kb_used_data":95368,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93641380,"statfs":{"total":96636764160,"available":95888773120,"internally_reserved":0,"allocated":97656832,"data_stored":97434316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608016,"kb_used_data":94592,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93763824,"statfs":{"total":96636764160,"available":96014155776,"internally_reserved":0,"allocated":96862208,"data_stored":96639692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525725394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729440,"kb_used_data":93904,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93642400,"statfs":{"total":96636764160,"available":95889817600,"internally_reserved":0,"allocated":96157696,"data_stored":95935180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650768079},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95326208,"data_stored":95326208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95760384,"data_stored":95760384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97132544,"data_stored":97132544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98443264,"data_stored":98443264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:05.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:06.176 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:06.176 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:06.209 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-10T20:29:06.089624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188317544,"num_objects":22992,"num_object_clones":0,"num_object_copies":91968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142226,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160677,"ondisk_log_size":160677,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779368,"kb_used_data":372172,"kb_used_omap":14,"kb_used_meta":2407089,"kb_avail":374707992,"statfs":{"total":386547056640,"available":383700983808,"internally_reserved":0,"allocated":381104128,"data_stored":380214064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2464859702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,10,6,23,14,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21053440,"num_objects":-2570,"num_object_clones":0,"num_object_copies":-10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":440,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003773"},"pg_stats":[{"pgid":"1.f","version":"42'10741","reported_seq":11027,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.394311+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:04.394311+0000","last_peered":"2024-09-10T20:29:04.394311+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:04.394311+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11313152,"num_objects":1381,"num_object_clones":0,"num_object_copies":5524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10768","reported_seq":10905,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.369112+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:02.369112+0000","last_peered":"2024-09-10T20:29:02.369112+0000","last_clean":"2024-09-10T20:29:02.369112+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:02.369112+0000","last_undegraded":"2024-09-10T20:29:02.369112+0000","last_fullsized":"2024-09-10T20:29:02.369112+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12156928,"num_objects":1484,"num_object_clones":0,"num_object_copies":5936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10716","reported_seq":10924,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:06.008718+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:06.008718+0000","last_peered":"2024-09-10T20:29:06.008718+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:06.008718+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11663976,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10715,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10889","reported_seq":15061,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.635891+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:02.635891+0000","last_peered":"2024-09-10T20:29:02.635891+0000","last_clean":"2024-09-10T20:29:02.635891+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:02.635891+0000","last_undegraded":"2024-09-10T20:29:02.635891+0000","last_fullsized":"2024-09-10T20:29:02.635891+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427264,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10712","reported_seq":11037,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.315900+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:04.315900+0000","last_peered":"2024-09-10T20:29:04.315900+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:04.315900+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11932832,"num_objects":1457,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10706","reported_seq":10933,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.284736+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:04.284736+0000","last_peered":"2024-09-10T20:29:04.284736+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:04.284736+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10655","reported_seq":10863,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:02.807463+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:02.807463+0000","last_peered":"2024-09-10T20:29:02.807463+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:02.807463+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11588768,"num_objects":1415,"num_object_clones":0,"num_object_copies":5660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10803","reported_seq":10976,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:02.947183+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:02.947183+0000","last_peered":"2024-09-10T20:29:02.947183+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:02.947183+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203168,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10595","reported_seq":14572,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:04.295534+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:04.295534+0000","last_peered":"2024-09-10T20:29:04.295534+0000","last_clean":"2024-09-10T20:29:04.295534+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:04.295534+0000","last_undegraded":"2024-09-10T20:29:04.295534+0000","last_fullsized":"2024-09-10T20:29:04.295534+0000","mapping_epoch":33,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11516496,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10708","reported_seq":14683,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:06.031081+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:06.031081+0000","last_peered":"2024-09-10T20:29:06.031081+0000","last_clean":"2024-09-10T20:29:06.031081+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:06.031081+0000","last_undegraded":"2024-09-10T20:29:06.031081+0000","last_fullsized":"2024-09-10T20:29:06.031081+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11747328,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10659","reported_seq":10937,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.316997+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:04.316997+0000","last_peered":"2024-09-10T20:29:04.316997+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:04.316997+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11906800,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10656,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-402}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10703","reported_seq":11006,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.172651+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:04.172651+0000","last_peered":"2024-09-10T20:29:04.172651+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:04.172651+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11975248,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-398}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10458","reported_seq":10646,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.978631+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:05.978631+0000","last_peered":"2024-09-10T20:29:05.978631+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:05.978631+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11246160,"num_objects":1373,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10680","reported_seq":14031,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.316807+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:04.316807+0000","last_peered":"2024-09-10T20:29:04.316807+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:04.316807+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11580576,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1640,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11035","reported_seq":11245,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.950144+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:05.950144+0000","last_peered":"2024-09-10T20:29:05.950144+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:05.950144+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11580576,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10649","reported_seq":14747,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.562983+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:02.562983+0000","last_peered":"2024-09-10T20:29:02.562983+0000","last_clean":"2024-09-10T20:29:02.562983+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:02.562983+0000","last_undegraded":"2024-09-10T20:29:02.562983+0000","last_fullsized":"2024-09-10T20:29:02.562983+0000","mapping_epoch":41,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12075008,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":188317544,"num_objects":22992,"num_object_clones":0,"num_object_copies":91968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142226,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377933824,"data_stored":377933824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160677,"ondisk_log_size":160677,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728200,"kb_used_data":92572,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93643640,"statfs":{"total":96636764160,"available":95891087360,"internally_reserved":0,"allocated":94793728,"data_stored":94571212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,9,4,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713712,"kb_used_data":91104,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93658128,"statfs":{"total":96636764160,"available":95905923072,"internally_reserved":0,"allocated":93290496,"data_stored":93067980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608016,"kb_used_data":94592,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93763824,"statfs":{"total":96636764160,"available":96014155776,"internally_reserved":0,"allocated":96862208,"data_stored":96639692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525725394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729440,"kb_used_data":93904,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93642400,"statfs":{"total":96636764160,"available":95889817600,"internally_reserved":0,"allocated":96157696,"data_stored":95935180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650768079},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95326208,"data_stored":95326208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95760384,"data_stored":95760384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92733440,"data_stored":92733440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94113792,"data_stored":94113792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:06.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:06.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10741", 'reported_seq': 11027, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:04.394311+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:04.394311+0000', 'last_peered': '2024-09-10T20:29:04.394311+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:04.394311+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11313152, 'num_objects': 1381, 'num_object_clones': 0, 'num_object_copies': 5524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10741, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:06.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10716", 'reported_seq': 10924, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:06.008718+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:06.008718+0000', 'last_peered': '2024-09-10T20:29:06.008718+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:06.008718+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11663976, 'num_objects': 1425, 'num_object_clones': 0, 'num_object_copies': 5700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1425, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10715, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -483}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10712", 'reported_seq': 11037, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:04.315900+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:04.315900+0000', 'last_peered': '2024-09-10T20:29:04.315900+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:04.315900+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11932832, 'num_objects': 1457, 'num_object_clones': 0, 'num_object_copies': 5828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10710, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -398}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10706", 'reported_seq': 10933, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:04.284736+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:04.284736+0000', 'last_peered': '2024-09-10T20:29:04.284736+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:04.284736+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11403264, 'num_objects': 1392, 'num_object_clones': 0, 'num_object_copies': 5568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10706, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10655", 'reported_seq': 10863, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:02.807463+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:29:02.807463+0000', 'last_peered': '2024-09-10T20:29:02.807463+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:02.807463+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11588768, 'num_objects': 1415, 'num_object_clones': 0, 'num_object_copies': 5660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10653, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -478}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10803", 'reported_seq': 10976, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:02.947183+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:02.947183+0000', 'last_peered': '2024-09-10T20:29:02.947183+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:02.947183+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12203168, 'num_objects': 1490, 'num_object_clones': 0, 'num_object_copies': 5960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10801, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:06.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10659", 'reported_seq': 10937, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:04.316997+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:04.316997+0000', 'last_peered': '2024-09-10T20:29:04.316997+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:04.316997+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11906800, 'num_objects': 1454, 'num_object_clones': 0, 'num_object_copies': 5816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10656, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -402}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10703", 'reported_seq': 11006, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:04.172651+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:04.172651+0000', 'last_peered': '2024-09-10T20:29:04.172651+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:04.172651+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11975248, 'num_objects': 1462, 'num_object_clones': 0, 'num_object_copies': 5848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10702, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10458", 'reported_seq': 10646, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:05.978631+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:05.978631+0000', 'last_peered': '2024-09-10T20:29:05.978631+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:05.978631+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11246160, 'num_objects': 1373, 'num_object_clones': 0, 'num_object_copies': 5492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10457, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10680", 'reported_seq': 14031, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:04.316807+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:29:04.316807+0000', 'last_peered': '2024-09-10T20:29:04.316807+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:29:04.316807+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11580576, 'num_objects': 1414, 'num_object_clones': 0, 'num_object_copies': 5656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10678, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1640, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:06.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'11035", 'reported_seq': 11245, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:05.950144+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:05.950144+0000', 'last_peered': '2024-09-10T20:29:05.950144+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:05.950144+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11580576, 'num_objects': 1414, 'num_object_clones': 0, 'num_object_copies': 5656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11033, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:06.213 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:06.984 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:06.984 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19248/91968 objects degraded (20.929%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":635,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22992,"data_bytes":188317544,"bytes_used":2846072832,"bytes_avail":383700983808,"bytes_total":386547056640,"degraded_objects":19248,"degraded_total":91968,"degraded_ratio":0.20929018789144049,"recovering_objects_per_sec":36,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":440,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":214},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:07.007 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:07.007 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:07.684 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:07.684 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:07.710 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-10T20:29:06.089624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188317544,"num_objects":22992,"num_object_clones":0,"num_object_copies":91968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142226,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160677,"ondisk_log_size":160677,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779368,"kb_used_data":372172,"kb_used_omap":14,"kb_used_meta":2407089,"kb_avail":374707992,"statfs":{"total":386547056640,"available":383700983808,"internally_reserved":0,"allocated":381104128,"data_stored":380214064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2464859702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,10,6,23,14,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21053440,"num_objects":-2570,"num_object_clones":0,"num_object_copies":-10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":440,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003773"},"pg_stats":[{"pgid":"1.f","version":"42'10741","reported_seq":11027,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.394311+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:04.394311+0000","last_peered":"2024-09-10T20:29:04.394311+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:04.394311+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11313152,"num_objects":1381,"num_object_clones":0,"num_object_copies":5524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10768","reported_seq":10905,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.369112+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:02.369112+0000","last_peered":"2024-09-10T20:29:02.369112+0000","last_clean":"2024-09-10T20:29:02.369112+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:02.369112+0000","last_undegraded":"2024-09-10T20:29:02.369112+0000","last_fullsized":"2024-09-10T20:29:02.369112+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12156928,"num_objects":1484,"num_object_clones":0,"num_object_copies":5936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10716","reported_seq":10924,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:06.008718+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:06.008718+0000","last_peered":"2024-09-10T20:29:06.008718+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:06.008718+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11663976,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10715,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10889","reported_seq":15061,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.635891+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:02.635891+0000","last_peered":"2024-09-10T20:29:02.635891+0000","last_clean":"2024-09-10T20:29:02.635891+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:02.635891+0000","last_undegraded":"2024-09-10T20:29:02.635891+0000","last_fullsized":"2024-09-10T20:29:02.635891+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427264,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10712","reported_seq":11037,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.315900+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:04.315900+0000","last_peered":"2024-09-10T20:29:04.315900+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:04.315900+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11932832,"num_objects":1457,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10706","reported_seq":10933,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.284736+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:04.284736+0000","last_peered":"2024-09-10T20:29:04.284736+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:04.284736+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10655","reported_seq":10863,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:02.807463+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:02.807463+0000","last_peered":"2024-09-10T20:29:02.807463+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:02.807463+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11588768,"num_objects":1415,"num_object_clones":0,"num_object_copies":5660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10803","reported_seq":10976,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:02.947183+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:02.947183+0000","last_peered":"2024-09-10T20:29:02.947183+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:02.947183+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203168,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10595","reported_seq":14572,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:04.295534+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:04.295534+0000","last_peered":"2024-09-10T20:29:04.295534+0000","last_clean":"2024-09-10T20:29:04.295534+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:04.295534+0000","last_undegraded":"2024-09-10T20:29:04.295534+0000","last_fullsized":"2024-09-10T20:29:04.295534+0000","mapping_epoch":33,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11516496,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10708","reported_seq":14683,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:06.031081+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:06.031081+0000","last_peered":"2024-09-10T20:29:06.031081+0000","last_clean":"2024-09-10T20:29:06.031081+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:06.031081+0000","last_undegraded":"2024-09-10T20:29:06.031081+0000","last_fullsized":"2024-09-10T20:29:06.031081+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11747328,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10659","reported_seq":10937,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.316997+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:04.316997+0000","last_peered":"2024-09-10T20:29:04.316997+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:04.316997+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11906800,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10656,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-402}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10703","reported_seq":11006,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.172651+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:04.172651+0000","last_peered":"2024-09-10T20:29:04.172651+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:04.172651+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11975248,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-398}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10458","reported_seq":10646,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.978631+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:05.978631+0000","last_peered":"2024-09-10T20:29:05.978631+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:05.978631+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11246160,"num_objects":1373,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10680","reported_seq":14031,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:04.316807+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:04.316807+0000","last_peered":"2024-09-10T20:29:04.316807+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:04.316807+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11580576,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1640,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11035","reported_seq":11245,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.950144+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:05.950144+0000","last_peered":"2024-09-10T20:29:05.950144+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:05.950144+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11580576,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10649","reported_seq":14747,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:02.562983+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:02.562983+0000","last_peered":"2024-09-10T20:29:02.562983+0000","last_clean":"2024-09-10T20:29:02.562983+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:02.562983+0000","last_undegraded":"2024-09-10T20:29:02.562983+0000","last_fullsized":"2024-09-10T20:29:02.562983+0000","mapping_epoch":41,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12075008,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":188317544,"num_objects":22992,"num_object_clones":0,"num_object_copies":91968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142226,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377933824,"data_stored":377933824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160677,"ondisk_log_size":160677,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728200,"kb_used_data":92572,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93643640,"statfs":{"total":96636764160,"available":95891087360,"internally_reserved":0,"allocated":94793728,"data_stored":94571212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,9,4,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713712,"kb_used_data":91104,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93658128,"statfs":{"total":96636764160,"available":95905923072,"internally_reserved":0,"allocated":93290496,"data_stored":93067980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608016,"kb_used_data":94592,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93763824,"statfs":{"total":96636764160,"available":96014155776,"internally_reserved":0,"allocated":96862208,"data_stored":96639692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525725394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729440,"kb_used_data":93904,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93642400,"statfs":{"total":96636764160,"available":95889817600,"internally_reserved":0,"allocated":96157696,"data_stored":95935180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650768079},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95326208,"data_stored":95326208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95760384,"data_stored":95760384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92733440,"data_stored":92733440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94113792,"data_stored":94113792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:10.713 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:11.314 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:11.314 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:11.345 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-10T20:29:10.091114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182206312,"num_objects":22246,"num_object_clones":0,"num_object_copies":88984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142972,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729636,"kb_used_data":359892,"kb_used_omap":14,"kb_used_meta":2369649,"kb_avail":374757724,"statfs":{"total":386547056640,"available":383751909376,"internally_reserved":0,"allocated":368529408,"data_stored":367639344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2426521142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,12,20,14,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20725760,"num_objects":-2530,"num_object_clones":0,"num_object_copies":-10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003895"},"pg_stats":[{"pgid":"1.f","version":"42'10804","reported_seq":11086,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.243072+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:09.243072+0000","last_peered":"2024-09-10T20:29:09.243072+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:09.243072+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10797056,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10827","reported_seq":10964,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.258732+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:07.258732+0000","last_peered":"2024-09-10T20:29:07.258732+0000","last_clean":"2024-09-10T20:29:07.258732+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:07.258732+0000","last_undegraded":"2024-09-10T20:29:07.258732+0000","last_fullsized":"2024-09-10T20:29:07.258732+0000","mapping_epoch":35,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11673600,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10716","reported_seq":10924,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:06.008718+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:06.008718+0000","last_peered":"2024-09-10T20:29:06.008718+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:06.008718+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11663976,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10715,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10946","reported_seq":15119,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.622596+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:07.622596+0000","last_peered":"2024-09-10T20:29:07.622596+0000","last_clean":"2024-09-10T20:29:07.622596+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:07.622596+0000","last_undegraded":"2024-09-10T20:29:07.622596+0000","last_fullsized":"2024-09-10T20:29:07.622596+0000","mapping_epoch":37,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":1460,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10777","reported_seq":11102,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.337415+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:09.337415+0000","last_peered":"2024-09-10T20:29:09.337415+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:09.337415+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400352,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10774","reported_seq":11001,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.283989+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:09.283989+0000","last_peered":"2024-09-10T20:29:09.283989+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:09.283989+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10846208,"num_objects":1324,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10704","reported_seq":10911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:07.919616+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:07.919616+0000","last_peered":"2024-09-10T20:29:07.919616+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:07.919616+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11187360,"num_objects":1366,"num_object_clones":0,"num_object_copies":5464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-527}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10865","reported_seq":11038,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:07.918774+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:07.918774+0000","last_peered":"2024-09-10T20:29:07.918774+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:07.918774+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11695264,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10651","reported_seq":14628,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:09.427229+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:09.427229+0000","last_peered":"2024-09-10T20:29:09.427229+0000","last_clean":"2024-09-10T20:29:09.427229+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:09.427229+0000","last_undegraded":"2024-09-10T20:29:09.427229+0000","last_fullsized":"2024-09-10T20:29:09.427229+0000","mapping_epoch":33,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11057744,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10708","reported_seq":14683,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:06.031081+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:06.031081+0000","last_peered":"2024-09-10T20:29:06.031081+0000","last_clean":"2024-09-10T20:29:06.031081+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:06.031081+0000","last_undegraded":"2024-09-10T20:29:06.031081+0000","last_fullsized":"2024-09-10T20:29:06.031081+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11747328,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10722","reported_seq":11000,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.305297+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:09.305297+0000","last_peered":"2024-09-10T20:29:09.305297+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:09.305297+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11390704,"num_objects":1391,"num_object_clones":0,"num_object_copies":5564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10773","reported_seq":11075,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.463984+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:09.463984+0000","last_peered":"2024-09-10T20:29:09.463984+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:09.463984+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11401808,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10458","reported_seq":10646,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.978631+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:05.978631+0000","last_peered":"2024-09-10T20:29:05.978631+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:05.978631+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11246160,"num_objects":1373,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10747","reported_seq":14203,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.474594+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:09.474594+0000","last_peered":"2024-09-10T20:29:09.474594+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:09.474594+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11031712,"num_objects":1347,"num_object_clones":0,"num_object_copies":5388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11035","reported_seq":11245,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.950144+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:05.950144+0000","last_peered":"2024-09-10T20:29:05.950144+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:05.950144+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11580576,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10716","reported_seq":14816,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.589759+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:07.589759+0000","last_peered":"2024-09-10T20:29:07.589759+0000","last_clean":"2024-09-10T20:29:07.589759+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:07.589759+0000","last_undegraded":"2024-09-10T20:29:07.589759+0000","last_fullsized":"2024-09-10T20:29:07.589759+0000","mapping_epoch":41,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11526144,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":182206312,"num_objects":22246,"num_object_clones":0,"num_object_copies":88984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142972,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365346816,"data_stored":365346816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711736,"kb_used_data":88604,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93660104,"statfs":{"total":96636764160,"available":95907946496,"internally_reserved":0,"allocated":90730496,"data_stored":90507980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713712,"kb_used_data":91104,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93658128,"statfs":{"total":96636764160,"available":95905923072,"internally_reserved":0,"allocated":93290496,"data_stored":93067980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591320,"kb_used_data":90376,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93780520,"statfs":{"total":96636764160,"available":96031252480,"internally_reserved":0,"allocated":92545024,"data_stored":92322508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512945874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712868,"kb_used_data":89808,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93658972,"statfs":{"total":96636764160,"available":95906787328,"internally_reserved":0,"allocated":91963392,"data_stored":91740876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91148288,"data_stored":91148288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91578368,"data_stored":91578368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92733440,"data_stored":92733440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":89886720,"data_stored":89886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:11.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:11.988 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:12.010 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-10T20:29:10.091114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182206312,"num_objects":22246,"num_object_clones":0,"num_object_copies":88984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142972,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729636,"kb_used_data":359892,"kb_used_omap":14,"kb_used_meta":2369649,"kb_avail":374757724,"statfs":{"total":386547056640,"available":383751909376,"internally_reserved":0,"allocated":368529408,"data_stored":367639344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2426521142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,12,20,14,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20725760,"num_objects":-2530,"num_object_clones":0,"num_object_copies":-10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003895"},"pg_stats":[{"pgid":"1.f","version":"42'10804","reported_seq":11086,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.243072+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:09.243072+0000","last_peered":"2024-09-10T20:29:09.243072+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:09.243072+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10797056,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10827","reported_seq":10964,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.258732+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:07.258732+0000","last_peered":"2024-09-10T20:29:07.258732+0000","last_clean":"2024-09-10T20:29:07.258732+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:07.258732+0000","last_undegraded":"2024-09-10T20:29:07.258732+0000","last_fullsized":"2024-09-10T20:29:07.258732+0000","mapping_epoch":35,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11673600,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10716","reported_seq":10924,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:06.008718+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:06.008718+0000","last_peered":"2024-09-10T20:29:06.008718+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:06.008718+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11663976,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10715,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10946","reported_seq":15119,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.622596+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:07.622596+0000","last_peered":"2024-09-10T20:29:07.622596+0000","last_clean":"2024-09-10T20:29:07.622596+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:07.622596+0000","last_undegraded":"2024-09-10T20:29:07.622596+0000","last_fullsized":"2024-09-10T20:29:07.622596+0000","mapping_epoch":37,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":1460,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10777","reported_seq":11102,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.337415+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:09.337415+0000","last_peered":"2024-09-10T20:29:09.337415+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:09.337415+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400352,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10774","reported_seq":11001,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.283989+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:09.283989+0000","last_peered":"2024-09-10T20:29:09.283989+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:09.283989+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10846208,"num_objects":1324,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10704","reported_seq":10911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:07.919616+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:07.919616+0000","last_peered":"2024-09-10T20:29:07.919616+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:07.919616+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11187360,"num_objects":1366,"num_object_clones":0,"num_object_copies":5464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-527}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10865","reported_seq":11038,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:07.918774+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:07.918774+0000","last_peered":"2024-09-10T20:29:07.918774+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:07.918774+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11695264,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10651","reported_seq":14628,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:09.427229+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:09.427229+0000","last_peered":"2024-09-10T20:29:09.427229+0000","last_clean":"2024-09-10T20:29:09.427229+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:09.427229+0000","last_undegraded":"2024-09-10T20:29:09.427229+0000","last_fullsized":"2024-09-10T20:29:09.427229+0000","mapping_epoch":33,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11057744,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10708","reported_seq":14683,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:06.031081+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:06.031081+0000","last_peered":"2024-09-10T20:29:06.031081+0000","last_clean":"2024-09-10T20:29:06.031081+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:06.031081+0000","last_undegraded":"2024-09-10T20:29:06.031081+0000","last_fullsized":"2024-09-10T20:29:06.031081+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11747328,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10722","reported_seq":11000,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.305297+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:09.305297+0000","last_peered":"2024-09-10T20:29:09.305297+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:09.305297+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11390704,"num_objects":1391,"num_object_clones":0,"num_object_copies":5564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10773","reported_seq":11075,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.463984+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:09.463984+0000","last_peered":"2024-09-10T20:29:09.463984+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:09.463984+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11401808,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10458","reported_seq":10646,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.978631+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:05.978631+0000","last_peered":"2024-09-10T20:29:05.978631+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:05.978631+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11246160,"num_objects":1373,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10747","reported_seq":14203,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.474594+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:09.474594+0000","last_peered":"2024-09-10T20:29:09.474594+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:09.474594+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11031712,"num_objects":1347,"num_object_clones":0,"num_object_copies":5388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11035","reported_seq":11245,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:05.950144+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:05.950144+0000","last_peered":"2024-09-10T20:29:05.950144+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:05.950144+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11580576,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10716","reported_seq":14816,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.589759+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:07.589759+0000","last_peered":"2024-09-10T20:29:07.589759+0000","last_clean":"2024-09-10T20:29:07.589759+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:07.589759+0000","last_undegraded":"2024-09-10T20:29:07.589759+0000","last_fullsized":"2024-09-10T20:29:07.589759+0000","mapping_epoch":41,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11526144,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":182206312,"num_objects":22246,"num_object_clones":0,"num_object_copies":88984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142972,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365346816,"data_stored":365346816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711736,"kb_used_data":88604,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93660104,"statfs":{"total":96636764160,"available":95907946496,"internally_reserved":0,"allocated":90730496,"data_stored":90507980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713712,"kb_used_data":91104,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93658128,"statfs":{"total":96636764160,"available":95905923072,"internally_reserved":0,"allocated":93290496,"data_stored":93067980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591320,"kb_used_data":90376,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93780520,"statfs":{"total":96636764160,"available":96031252480,"internally_reserved":0,"allocated":92545024,"data_stored":92322508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512945874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712868,"kb_used_data":89808,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93658972,"statfs":{"total":96636764160,"available":95906787328,"internally_reserved":0,"allocated":91963392,"data_stored":91740876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91148288,"data_stored":91148288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91578368,"data_stored":91578368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92733440,"data_stored":92733440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":89886720,"data_stored":89886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:12.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:12.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10804", 'reported_seq': 11086, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:09.243072+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:09.243072+0000', 'last_peered': '2024-09-10T20:29:09.243072+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:09.243072+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10797056, 'num_objects': 1318, 'num_object_clones': 0, 'num_object_copies': 5272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10804, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -537}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:12.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:12.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10716", 'reported_seq': 10924, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:06.008718+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:06.008718+0000', 'last_peered': '2024-09-10T20:29:06.008718+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:06.008718+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11663976, 'num_objects': 1425, 'num_object_clones': 0, 'num_object_copies': 5700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1425, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10715, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -483}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:12.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:12.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10777", 'reported_seq': 11102, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:09.337415+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:09.337415+0000', 'last_peered': '2024-09-10T20:29:09.337415+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:09.337415+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11400352, 'num_objects': 1392, 'num_object_clones': 0, 'num_object_copies': 5568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10775, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -463}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10774", 'reported_seq': 11001, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:09.283989+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:09.283989+0000', 'last_peered': '2024-09-10T20:29:09.283989+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:09.283989+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10846208, 'num_objects': 1324, 'num_object_clones': 0, 'num_object_copies': 5296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10774, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -533}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10704", 'reported_seq': 10911, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:07.919616+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:29:07.919616+0000', 'last_peered': '2024-09-10T20:29:07.919616+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:07.919616+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11187360, 'num_objects': 1366, 'num_object_clones': 0, 'num_object_copies': 5464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10702, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -527}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10865", 'reported_seq': 11038, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:07.918774+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:07.918774+0000', 'last_peered': '2024-09-10T20:29:07.918774+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:07.918774+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11695264, 'num_objects': 1428, 'num_object_clones': 0, 'num_object_copies': 5712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10863, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10722", 'reported_seq': 11000, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:09.305297+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:09.305297+0000', 'last_peered': '2024-09-10T20:29:09.305297+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:09.305297+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11390704, 'num_objects': 1391, 'num_object_clones': 0, 'num_object_copies': 5564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10719, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:12.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10773", 'reported_seq': 11075, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:09.463984+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:09.463984+0000', 'last_peered': '2024-09-10T20:29:09.463984+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:09.463984+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11401808, 'num_objects': 1392, 'num_object_clones': 0, 'num_object_copies': 5568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10772, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:12.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:12.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10458", 'reported_seq': 10646, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:05.978631+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:05.978631+0000', 'last_peered': '2024-09-10T20:29:05.978631+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:05.978631+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11246160, 'num_objects': 1373, 'num_object_clones': 0, 'num_object_copies': 5492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10457, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:12.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:29:12.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10747", 'reported_seq': 14203, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:09.474594+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:29:09.474594+0000', 'last_peered': '2024-09-10T20:29:09.474594+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:29:09.474594+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11031712, 'num_objects': 1347, 'num_object_clones': 0, 'num_object_copies': 5388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 286, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1347, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10745, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1693, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:12.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:12.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'11035", 'reported_seq': 11245, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:05.950144+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:05.950144+0000', 'last_peered': '2024-09-10T20:29:05.950144+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:05.950144+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11580576, 'num_objects': 1414, 'num_object_clones': 0, 'num_object_copies': 5656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11033, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:12.013 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:12.778 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:12.778 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19195/88984 objects degraded (21.571%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":641,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22246,"data_bytes":182206312,"bytes_used":2795147264,"bytes_avail":383751909376,"bytes_total":386547056640,"degraded_objects":19195,"degraded_total":88984,"degraded_ratio":0.21571293715724174,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:12.791 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:12.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:13.357 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:13.358 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:13.380 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-10T20:29:12.091955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179863400,"num_objects":21960,"num_object_clones":0,"num_object_copies":87840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143258,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729736,"kb_used_data":355832,"kb_used_omap":14,"kb_used_meta":2373809,"kb_avail":374757624,"statfs":{"total":386547056640,"available":383751806976,"internally_reserved":0,"allocated":364371968,"data_stored":363481904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2430780982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,8,12,19,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19464192,"num_objects":-2376,"num_object_clones":0,"num_object_copies":-9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004075"},"pg_stats":[{"pgid":"1.f","version":"42'10804","reported_seq":11086,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.243072+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:09.243072+0000","last_peered":"2024-09-10T20:29:09.243072+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:09.243072+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10797056,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10827","reported_seq":10964,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.258732+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:07.258732+0000","last_peered":"2024-09-10T20:29:07.258732+0000","last_clean":"2024-09-10T20:29:07.258732+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:07.258732+0000","last_undegraded":"2024-09-10T20:29:07.258732+0000","last_fullsized":"2024-09-10T20:29:07.258732+0000","mapping_epoch":35,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11673600,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10799","reported_seq":11007,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.919668+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:10.919668+0000","last_peered":"2024-09-10T20:29:10.919668+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:10.919668+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10984040,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10798,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'10946","reported_seq":15119,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.622596+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:07.622596+0000","last_peered":"2024-09-10T20:29:07.622596+0000","last_clean":"2024-09-10T20:29:07.622596+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:07.622596+0000","last_undegraded":"2024-09-10T20:29:07.622596+0000","last_fullsized":"2024-09-10T20:29:07.622596+0000","mapping_epoch":37,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":1460,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10777","reported_seq":11102,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.337415+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:09.337415+0000","last_peered":"2024-09-10T20:29:09.337415+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:09.337415+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400352,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10774","reported_seq":11001,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.283989+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:09.283989+0000","last_peered":"2024-09-10T20:29:09.283989+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:09.283989+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10846208,"num_objects":1324,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10704","reported_seq":10911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:07.919616+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:07.919616+0000","last_peered":"2024-09-10T20:29:07.919616+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:07.919616+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11187360,"num_objects":1366,"num_object_clones":0,"num_object_copies":5464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-527}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10865","reported_seq":11038,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:07.918774+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:07.918774+0000","last_peered":"2024-09-10T20:29:07.918774+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:07.918774+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11695264,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10651","reported_seq":14628,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:09.427229+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:09.427229+0000","last_peered":"2024-09-10T20:29:09.427229+0000","last_clean":"2024-09-10T20:29:09.427229+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:09.427229+0000","last_undegraded":"2024-09-10T20:29:09.427229+0000","last_fullsized":"2024-09-10T20:29:09.427229+0000","mapping_epoch":33,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11057744,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10780","reported_seq":14755,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:11.133151+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:11.133151+0000","last_peered":"2024-09-10T20:29:11.133151+0000","last_clean":"2024-09-10T20:29:11.133151+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:11.133151+0000","last_undegraded":"2024-09-10T20:29:11.133151+0000","last_fullsized":"2024-09-10T20:29:11.133151+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11157504,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10722","reported_seq":11000,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.305297+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:09.305297+0000","last_peered":"2024-09-10T20:29:09.305297+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:09.305297+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11390704,"num_objects":1391,"num_object_clones":0,"num_object_copies":5564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10773","reported_seq":11075,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.463984+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:09.463984+0000","last_peered":"2024-09-10T20:29:09.463984+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:09.463984+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11401808,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10532","reported_seq":10720,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.932746+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:10.932746+0000","last_peered":"2024-09-10T20:29:10.932746+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:10.932746+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639952,"num_objects":1299,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10747","reported_seq":14203,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:09.474594+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:09.474594+0000","last_peered":"2024-09-10T20:29:09.474594+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:09.474594+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11031712,"num_objects":1347,"num_object_clones":0,"num_object_copies":5388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11092","reported_seq":11302,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.993642+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:10.993642+0000","last_peered":"2024-09-10T20:29:10.993642+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:10.993642+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11113632,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11090,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10716","reported_seq":14816,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:07.589759+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:07.589759+0000","last_peered":"2024-09-10T20:29:07.589759+0000","last_clean":"2024-09-10T20:29:07.589759+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:07.589759+0000","last_undegraded":"2024-09-10T20:29:07.589759+0000","last_fullsized":"2024-09-10T20:29:07.589759+0000","mapping_epoch":41,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11526144,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179863400,"num_objects":21960,"num_object_clones":0,"num_object_copies":87840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143258,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":361054208,"data_stored":361054208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711736,"kb_used_data":88604,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93660104,"statfs":{"total":96636764160,"available":95907946496,"internally_reserved":0,"allocated":90730496,"data_stored":90507980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713812,"kb_used_data":87044,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93658028,"statfs":{"total":96636764160,"available":95905820672,"internally_reserved":0,"allocated":89133056,"data_stored":88910540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591320,"kb_used_data":90376,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93780520,"statfs":{"total":96636764160,"available":96031252480,"internally_reserved":0,"allocated":92545024,"data_stored":92322508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512945874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712868,"kb_used_data":89808,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93658972,"statfs":{"total":96636764160,"available":95906787328,"internally_reserved":0,"allocated":91963392,"data_stored":91740876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91148288,"data_stored":91148288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91578368,"data_stored":91578368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":88440832,"data_stored":88440832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":89886720,"data_stored":89886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:16.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:17.007 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:17.007 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:17.027 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-10T20:29:16.093617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173686632,"num_objects":21206,"num_object_clones":0,"num_object_copies":84824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729824,"kb_used_data":343456,"kb_used_omap":14,"kb_used_meta":2386289,"kb_avail":374757536,"statfs":{"total":386547056640,"available":383751716864,"internally_reserved":0,"allocated":351698944,"data_stored":350808880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2443560502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,1,6,18,13,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20815872,"num_objects":-2541,"num_object_clones":0,"num_object_copies":-10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2541,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004455"},"pg_stats":[{"pgid":"1.f","version":"42'10862","reported_seq":11149,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.431555+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:14.431555+0000","last_peered":"2024-09-10T20:29:14.431555+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:14.431555+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10321920,"num_objects":1260,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-595}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10903","reported_seq":11039,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:12.575085+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:12.575085+0000","last_peered":"2024-09-10T20:29:12.575085+0000","last_clean":"2024-09-10T20:29:12.575085+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:12.575085+0000","last_undegraded":"2024-09-10T20:29:12.575085+0000","last_fullsized":"2024-09-10T20:29:12.575085+0000","mapping_epoch":35,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11051008,"num_objects":1349,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7344,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10799","reported_seq":11007,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.919668+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:10.919668+0000","last_peered":"2024-09-10T20:29:10.919668+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:10.919668+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10984040,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10798,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11005","reported_seq":15179,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:12.669316+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:12.669316+0000","last_peered":"2024-09-10T20:29:12.669316+0000","last_clean":"2024-09-10T20:29:12.669316+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:12.669316+0000","last_undegraded":"2024-09-10T20:29:12.669316+0000","last_fullsized":"2024-09-10T20:29:12.669316+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11476992,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5265,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10842","reported_seq":11167,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.435932+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:14.435932+0000","last_peered":"2024-09-10T20:29:14.435932+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:14.435932+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10867872,"num_objects":1327,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10843","reported_seq":11070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.434785+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:14.434785+0000","last_peered":"2024-09-10T20:29:14.434785+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:14.434785+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10280960,"num_objects":1255,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10764","reported_seq":10972,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:13.087133+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:13.087133+0000","last_peered":"2024-09-10T20:29:13.087133+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:13.087133+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10695840,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-587}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10926","reported_seq":11099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:13.044816+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:13.044816+0000","last_peered":"2024-09-10T20:29:13.044816+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:13.044816+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195552,"num_objects":1367,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10718","reported_seq":14695,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:14.373774+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:14.373774+0000","last_peered":"2024-09-10T20:29:14.373774+0000","last_clean":"2024-09-10T20:29:14.373774+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:14.373774+0000","last_undegraded":"2024-09-10T20:29:14.373774+0000","last_fullsized":"2024-09-10T20:29:14.373774+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10508880,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10780","reported_seq":14755,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:11.133151+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:11.133151+0000","last_peered":"2024-09-10T20:29:11.133151+0000","last_clean":"2024-09-10T20:29:11.133151+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:11.133151+0000","last_undegraded":"2024-09-10T20:29:11.133151+0000","last_fullsized":"2024-09-10T20:29:11.133151+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11157504,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10780","reported_seq":11058,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.357076+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:14.357076+0000","last_peered":"2024-09-10T20:29:14.357076+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:14.357076+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10915568,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10838","reported_seq":11141,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.519702+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:14.519702+0000","last_peered":"2024-09-10T20:29:14.519702+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:14.519702+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10869328,"num_objects":1327,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10532","reported_seq":10720,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.932746+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:10.932746+0000","last_peered":"2024-09-10T20:29:10.932746+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:10.932746+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639952,"num_objects":1299,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10806","reported_seq":14389,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.453746+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:14.453746+0000","last_peered":"2024-09-10T20:29:14.453746+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:14.453746+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10548384,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11092","reported_seq":11302,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.993642+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:10.993642+0000","last_peered":"2024-09-10T20:29:10.993642+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:10.993642+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11113632,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11090,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10773","reported_seq":14874,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:12.668649+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:12.668649+0000","last_peered":"2024-09-10T20:29:12.668649+0000","last_clean":"2024-09-10T20:29:12.668649+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:12.668649+0000","last_undegraded":"2024-09-10T20:29:12.668649+0000","last_fullsized":"2024-09-10T20:29:12.668649+0000","mapping_epoch":41,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11059200,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":173686632,"num_objects":21206,"num_object_clones":0,"num_object_copies":84824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348401664,"data_stored":348401664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711900,"kb_used_data":84620,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93659940,"statfs":{"total":96636764160,"available":95907778560,"internally_reserved":0,"allocated":86650880,"data_stored":86428364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,4,5,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713812,"kb_used_data":87044,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93658028,"statfs":{"total":96636764160,"available":95905820672,"internally_reserved":0,"allocated":89133056,"data_stored":88910540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591320,"kb_used_data":86216,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93780520,"statfs":{"total":96636764160,"available":96031252480,"internally_reserved":0,"allocated":88285184,"data_stored":88062668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,9,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712792,"kb_used_data":85576,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93659048,"statfs":{"total":96636764160,"available":95906865152,"internally_reserved":0,"allocated":87629824,"data_stored":87407308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86904832,"data_stored":86904832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87269376,"data_stored":87269376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":88440832,"data_stored":88440832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":85786624,"data_stored":85786624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:17.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:17.669 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:17.690 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-10T20:29:16.093617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173686632,"num_objects":21206,"num_object_clones":0,"num_object_copies":84824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729824,"kb_used_data":343456,"kb_used_omap":14,"kb_used_meta":2386289,"kb_avail":374757536,"statfs":{"total":386547056640,"available":383751716864,"internally_reserved":0,"allocated":351698944,"data_stored":350808880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2443560502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,1,6,18,13,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20815872,"num_objects":-2541,"num_object_clones":0,"num_object_copies":-10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2541,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004455"},"pg_stats":[{"pgid":"1.f","version":"42'10862","reported_seq":11149,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.431555+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:14.431555+0000","last_peered":"2024-09-10T20:29:14.431555+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:14.431555+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10321920,"num_objects":1260,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-595}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10903","reported_seq":11039,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:12.575085+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:12.575085+0000","last_peered":"2024-09-10T20:29:12.575085+0000","last_clean":"2024-09-10T20:29:12.575085+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:12.575085+0000","last_undegraded":"2024-09-10T20:29:12.575085+0000","last_fullsized":"2024-09-10T20:29:12.575085+0000","mapping_epoch":35,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11051008,"num_objects":1349,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7344,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10799","reported_seq":11007,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.919668+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:10.919668+0000","last_peered":"2024-09-10T20:29:10.919668+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:10.919668+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10984040,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10798,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11005","reported_seq":15179,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:12.669316+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:12.669316+0000","last_peered":"2024-09-10T20:29:12.669316+0000","last_clean":"2024-09-10T20:29:12.669316+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:12.669316+0000","last_undegraded":"2024-09-10T20:29:12.669316+0000","last_fullsized":"2024-09-10T20:29:12.669316+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11476992,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5265,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10842","reported_seq":11167,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.435932+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:14.435932+0000","last_peered":"2024-09-10T20:29:14.435932+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:14.435932+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10867872,"num_objects":1327,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10843","reported_seq":11070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.434785+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:14.434785+0000","last_peered":"2024-09-10T20:29:14.434785+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:14.434785+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10280960,"num_objects":1255,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10764","reported_seq":10972,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:13.087133+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:13.087133+0000","last_peered":"2024-09-10T20:29:13.087133+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:13.087133+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10695840,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-587}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10926","reported_seq":11099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:13.044816+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:13.044816+0000","last_peered":"2024-09-10T20:29:13.044816+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:13.044816+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195552,"num_objects":1367,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10718","reported_seq":14695,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:14.373774+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:14.373774+0000","last_peered":"2024-09-10T20:29:14.373774+0000","last_clean":"2024-09-10T20:29:14.373774+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:14.373774+0000","last_undegraded":"2024-09-10T20:29:14.373774+0000","last_fullsized":"2024-09-10T20:29:14.373774+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10508880,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10780","reported_seq":14755,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:11.133151+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:11.133151+0000","last_peered":"2024-09-10T20:29:11.133151+0000","last_clean":"2024-09-10T20:29:11.133151+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:11.133151+0000","last_undegraded":"2024-09-10T20:29:11.133151+0000","last_fullsized":"2024-09-10T20:29:11.133151+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11157504,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10780","reported_seq":11058,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.357076+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:14.357076+0000","last_peered":"2024-09-10T20:29:14.357076+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:14.357076+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10915568,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10838","reported_seq":11141,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.519702+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:14.519702+0000","last_peered":"2024-09-10T20:29:14.519702+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:14.519702+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10869328,"num_objects":1327,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10532","reported_seq":10720,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.932746+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:10.932746+0000","last_peered":"2024-09-10T20:29:10.932746+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:10.932746+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639952,"num_objects":1299,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10806","reported_seq":14389,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.453746+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:14.453746+0000","last_peered":"2024-09-10T20:29:14.453746+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:14.453746+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10548384,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11092","reported_seq":11302,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:10.993642+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:10.993642+0000","last_peered":"2024-09-10T20:29:10.993642+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:10.993642+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11113632,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11090,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10773","reported_seq":14874,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:12.668649+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:12.668649+0000","last_peered":"2024-09-10T20:29:12.668649+0000","last_clean":"2024-09-10T20:29:12.668649+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:12.668649+0000","last_undegraded":"2024-09-10T20:29:12.668649+0000","last_fullsized":"2024-09-10T20:29:12.668649+0000","mapping_epoch":41,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11059200,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":173686632,"num_objects":21206,"num_object_clones":0,"num_object_copies":84824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144012,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348401664,"data_stored":348401664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711900,"kb_used_data":84620,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93659940,"statfs":{"total":96636764160,"available":95907778560,"internally_reserved":0,"allocated":86650880,"data_stored":86428364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,4,5,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713812,"kb_used_data":87044,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93658028,"statfs":{"total":96636764160,"available":95905820672,"internally_reserved":0,"allocated":89133056,"data_stored":88910540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591320,"kb_used_data":86216,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93780520,"statfs":{"total":96636764160,"available":96031252480,"internally_reserved":0,"allocated":88285184,"data_stored":88062668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,9,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712792,"kb_used_data":85576,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93659048,"statfs":{"total":96636764160,"available":95906865152,"internally_reserved":0,"allocated":87629824,"data_stored":87407308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86904832,"data_stored":86904832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87269376,"data_stored":87269376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":88440832,"data_stored":88440832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":85786624,"data_stored":85786624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10862", 'reported_seq': 11149, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:14.431555+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:14.431555+0000', 'last_peered': '2024-09-10T20:29:14.431555+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:14.431555+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10321920, 'num_objects': 1260, 'num_object_clones': 0, 'num_object_copies': 5040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10862, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -595}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10799", 'reported_seq': 11007, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:10.919668+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:10.919668+0000', 'last_peered': '2024-09-10T20:29:10.919668+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:10.919668+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10984040, 'num_objects': 1342, 'num_object_clones': 0, 'num_object_copies': 5368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1342, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10798, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -566}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10842", 'reported_seq': 11167, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:14.435932+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:14.435932+0000', 'last_peered': '2024-09-10T20:29:14.435932+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:14.435932+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10867872, 'num_objects': 1327, 'num_object_clones': 0, 'num_object_copies': 5308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10840, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -528}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10843", 'reported_seq': 11070, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:14.434785+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:14.434785+0000', 'last_peered': '2024-09-10T20:29:14.434785+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:14.434785+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10280960, 'num_objects': 1255, 'num_object_clones': 0, 'num_object_copies': 5020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10843, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -602}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:17.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10764", 'reported_seq': 10972, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:13.087133+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:29:13.087133+0000', 'last_peered': '2024-09-10T20:29:13.087133+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:13.087133+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10695840, 'num_objects': 1306, 'num_object_clones': 0, 'num_object_copies': 5224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10762, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -587}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10926", 'reported_seq': 11099, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:13.044816+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:13.044816+0000', 'last_peered': '2024-09-10T20:29:13.044816+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:13.044816+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11195552, 'num_objects': 1367, 'num_object_clones': 0, 'num_object_copies': 5468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10924, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -633}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10780", 'reported_seq': 11058, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:14.357076+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:14.357076+0000', 'last_peered': '2024-09-10T20:29:14.357076+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:14.357076+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10915568, 'num_objects': 1333, 'num_object_clones': 0, 'num_object_copies': 5332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10777, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -523}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10838", 'reported_seq': 11141, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:14.519702+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:14.519702+0000', 'last_peered': '2024-09-10T20:29:14.519702+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:14.519702+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10869328, 'num_objects': 1327, 'num_object_clones': 0, 'num_object_copies': 5308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10837, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -533}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10532", 'reported_seq': 10720, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:10.932746+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:10.932746+0000', 'last_peered': '2024-09-10T20:29:10.932746+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:10.932746+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10639952, 'num_objects': 1299, 'num_object_clones': 0, 'num_object_copies': 5196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10531, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:29:17.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10806", 'reported_seq': 14389, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:14.453746+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:29:14.453746+0000', 'last_peered': '2024-09-10T20:29:14.453746+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:29:14.453746+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10548384, 'num_objects': 1288, 'num_object_clones': 0, 'num_object_copies': 5152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10804, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1756, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1065}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:17.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:17.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'11092", 'reported_seq': 11302, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:10.993642+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:10.993642+0000', 'last_peered': '2024-09-10T20:29:10.993642+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:10.993642+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11113632, 'num_objects': 1357, 'num_object_clones': 0, 'num_object_copies': 5428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11090, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -551}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:17.694 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:18.474 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19132/84824 objects degraded (22.555%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":647,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21206,"data_bytes":173686632,"bytes_used":2795339776,"bytes_avail":383751716864,"bytes_total":386547056640,"degraded_objects":19132,"degraded_total":84824,"degraded_ratio":0.22554937281901349,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":211},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:18.506 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:18.506 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:19.152 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:19.153 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:19.190 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-10T20:29:18.094426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169893736,"num_objects":20743,"num_object_clones":0,"num_object_copies":82972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144475,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730164,"kb_used_data":335476,"kb_used_omap":14,"kb_used_meta":2394609,"kb_avail":374757196,"statfs":{"total":386547056640,"available":383751368704,"internally_reserved":0,"allocated":343527424,"data_stored":342637360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2452080182},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,2,6,7,15,17,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18423808,"num_objects":-2249,"num_object_clones":0,"num_object_copies":-8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004801"},"pg_stats":[{"pgid":"1.f","version":"42'10862","reported_seq":11149,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.431555+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:14.431555+0000","last_peered":"2024-09-10T20:29:14.431555+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:14.431555+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10321920,"num_objects":1260,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-595}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10980","reported_seq":11117,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.824681+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:17.824681+0000","last_peered":"2024-09-10T20:29:17.824681+0000","last_clean":"2024-09-10T20:29:17.824681+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:17.824681+0000","last_undegraded":"2024-09-10T20:29:17.824681+0000","last_fullsized":"2024-09-10T20:29:17.824681+0000","mapping_epoch":35,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10855","reported_seq":11063,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:16.212340+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:16.212340+0000","last_peered":"2024-09-10T20:29:16.212340+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:16.212340+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10525288,"num_objects":1286,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10854,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-622}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11071","reported_seq":15244,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.798189+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:17.798189+0000","last_peered":"2024-09-10T20:29:17.798189+0000","last_clean":"2024-09-10T20:29:17.798189+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:17.798189+0000","last_undegraded":"2024-09-10T20:29:17.798189+0000","last_fullsized":"2024-09-10T20:29:17.798189+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10936320,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10842","reported_seq":11167,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.435932+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:14.435932+0000","last_peered":"2024-09-10T20:29:14.435932+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:14.435932+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10867872,"num_objects":1327,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10843","reported_seq":11070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.434785+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:14.434785+0000","last_peered":"2024-09-10T20:29:14.434785+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:14.434785+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10280960,"num_objects":1255,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10764","reported_seq":10972,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:13.087133+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:13.087133+0000","last_peered":"2024-09-10T20:29:13.087133+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:13.087133+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10695840,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-587}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10926","reported_seq":11099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:13.044816+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:13.044816+0000","last_peered":"2024-09-10T20:29:13.044816+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:13.044816+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195552,"num_objects":1367,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10718","reported_seq":14695,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:14.373774+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:14.373774+0000","last_peered":"2024-09-10T20:29:14.373774+0000","last_clean":"2024-09-10T20:29:14.373774+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:14.373774+0000","last_undegraded":"2024-09-10T20:29:14.373774+0000","last_fullsized":"2024-09-10T20:29:14.373774+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10508880,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10851","reported_seq":14823,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:16.258303+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:16.258303+0000","last_peered":"2024-09-10T20:29:16.258303+0000","last_clean":"2024-09-10T20:29:16.258303+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:16.258303+0000","last_undegraded":"2024-09-10T20:29:16.258303+0000","last_fullsized":"2024-09-10T20:29:16.258303+0000","mapping_epoch":39,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10575872,"num_objects":1291,"num_object_clones":0,"num_object_copies":5164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10780","reported_seq":11058,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.357076+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:14.357076+0000","last_peered":"2024-09-10T20:29:14.357076+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:14.357076+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10915568,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10838","reported_seq":11141,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.519702+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:14.519702+0000","last_peered":"2024-09-10T20:29:14.519702+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:14.519702+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10869328,"num_objects":1327,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10598","reported_seq":10786,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:16.136969+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:16.136969+0000","last_peered":"2024-09-10T20:29:16.136969+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:16.136969+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10099280,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-670}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10806","reported_seq":14389,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:14.453746+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:14.453746+0000","last_peered":"2024-09-10T20:29:14.453746+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:14.453746+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10548384,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11152","reported_seq":11362,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:16.110891+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:16.110891+0000","last_peered":"2024-09-10T20:29:16.110891+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:16.110891+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10622112,"num_objects":1297,"num_object_clones":0,"num_object_copies":5188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11150,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10840","reported_seq":14940,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.544525+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:17.544525+0000","last_peered":"2024-09-10T20:29:17.544525+0000","last_clean":"2024-09-10T20:29:17.544525+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:17.544525+0000","last_undegraded":"2024-09-10T20:29:17.544525+0000","last_fullsized":"2024-09-10T20:29:17.544525+0000","mapping_epoch":41,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10510336,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":169893736,"num_objects":20743,"num_object_clones":0,"num_object_copies":82972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144475,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339992576,"data_stored":339992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711900,"kb_used_data":84620,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93659940,"statfs":{"total":96636764160,"available":95907778560,"internally_reserved":0,"allocated":86650880,"data_stored":86428364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,4,5,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714040,"kb_used_data":83112,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93657800,"statfs":{"total":96636764160,"available":95905587200,"internally_reserved":0,"allocated":85106688,"data_stored":84884172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,0,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591432,"kb_used_data":82168,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93780408,"statfs":{"total":96636764160,"available":96031137792,"internally_reserved":0,"allocated":84140032,"data_stored":83917516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712792,"kb_used_data":85576,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93659048,"statfs":{"total":96636764160,"available":95906865152,"internally_reserved":0,"allocated":87629824,"data_stored":87407308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86904832,"data_stored":86904832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83042304,"data_stored":83042304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":84258816,"data_stored":84258816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":85786624,"data_stored":85786624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:22.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:22.831 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:22.851 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-10T20:29:22.096520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163077992,"num_objects":19911,"num_object_clones":0,"num_object_copies":79644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145307,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730368,"kb_used_data":323172,"kb_used_omap":14,"kb_used_meta":2407089,"kb_avail":374756992,"statfs":{"total":386547056640,"available":383751159808,"internally_reserved":0,"allocated":330928128,"data_stored":330038064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2464859702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,5,5,17,18,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19128320,"num_objects":-2335,"num_object_clones":0,"num_object_copies":-9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2335,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005406"},"pg_stats":[{"pgid":"1.f","version":"42'10923","reported_seq":11210,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.474954+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:19.474954+0000","last_peered":"2024-09-10T20:29:19.474954+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:19.474954+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9822208,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10923,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10980","reported_seq":11117,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.824681+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:17.824681+0000","last_peered":"2024-09-10T20:29:17.824681+0000","last_clean":"2024-09-10T20:29:17.824681+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:17.824681+0000","last_undegraded":"2024-09-10T20:29:17.824681+0000","last_fullsized":"2024-09-10T20:29:17.824681+0000","mapping_epoch":35,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10922","reported_seq":11130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.067576+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:21.067576+0000","last_peered":"2024-09-10T20:29:21.067576+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:21.067576+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9976424,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10921,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11071","reported_seq":15244,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.798189+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:17.798189+0000","last_peered":"2024-09-10T20:29:17.798189+0000","last_clean":"2024-09-10T20:29:17.798189+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:17.798189+0000","last_undegraded":"2024-09-10T20:29:17.798189+0000","last_fullsized":"2024-09-10T20:29:17.798189+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10936320,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10902","reported_seq":11227,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.490630+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:19.490630+0000","last_peered":"2024-09-10T20:29:19.490630+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:19.490630+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10376352,"num_objects":1267,"num_object_clones":0,"num_object_copies":5068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-588}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10904","reported_seq":11131,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.492376+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:19.492376+0000","last_peered":"2024-09-10T20:29:19.492376+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:19.492376+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9781248,"num_objects":1194,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-663}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10831","reported_seq":11038,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:18.046590+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:18.046590+0000","last_peered":"2024-09-10T20:29:18.046590+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:18.046590+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10146976,"num_objects":1239,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10989","reported_seq":11160,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:18.170859+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:18.170859+0000","last_peered":"2024-09-10T20:29:18.170859+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:18.170859+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679456,"num_objects":1304,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10773","reported_seq":14750,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:19.474706+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:19.474706+0000","last_peered":"2024-09-10T20:29:19.474706+0000","last_clean":"2024-09-10T20:29:19.474706+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:19.474706+0000","last_undegraded":"2024-09-10T20:29:19.474706+0000","last_fullsized":"2024-09-10T20:29:19.474706+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10058320,"num_objects":1228,"num_object_clones":0,"num_object_copies":4912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10915","reported_seq":14891,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:21.298875+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:21.298875+0000","last_peered":"2024-09-10T20:29:21.298875+0000","last_clean":"2024-09-10T20:29:21.298875+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:21.298875+0000","last_undegraded":"2024-09-10T20:29:21.298875+0000","last_fullsized":"2024-09-10T20:29:21.298875+0000","mapping_epoch":39,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10051584,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10855","reported_seq":11133,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.506932+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:19.506932+0000","last_peered":"2024-09-10T20:29:19.506932+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:19.506932+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10301168,"num_objects":1258,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10895","reported_seq":11198,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.547538+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:19.547538+0000","last_peered":"2024-09-10T20:29:19.547538+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:19.547538+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10402384,"num_objects":1270,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10664","reported_seq":10852,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.240165+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:21.240165+0000","last_peered":"2024-09-10T20:29:21.240165+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:21.240165+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9558608,"num_objects":1167,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10875","reported_seq":14569,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.540527+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:19.540527+0000","last_peered":"2024-09-10T20:29:19.540527+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:19.540527+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983136,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1812,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11219","reported_seq":11429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.193603+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:21.193603+0000","last_peered":"2024-09-10T20:29:21.193603+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:21.193603+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073248,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11217,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10840","reported_seq":14940,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.544525+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:17.544525+0000","last_peered":"2024-09-10T20:29:17.544525+0000","last_clean":"2024-09-10T20:29:17.544525+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:17.544525+0000","last_undegraded":"2024-09-10T20:29:17.544525+0000","last_fullsized":"2024-09-10T20:29:17.544525+0000","mapping_epoch":41,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10510336,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":163077992,"num_objects":19911,"num_object_clones":0,"num_object_copies":79644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145307,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327245824,"data_stored":327245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711976,"kb_used_data":80536,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93659864,"statfs":{"total":96636764160,"available":95907700736,"internally_reserved":0,"allocated":82468864,"data_stored":82246348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714000,"kb_used_data":78884,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93657840,"statfs":{"total":96636764160,"available":95905628160,"internally_reserved":0,"allocated":80777216,"data_stored":80554700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591432,"kb_used_data":82168,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93780408,"statfs":{"total":96636764160,"available":96031137792,"internally_reserved":0,"allocated":84140032,"data_stored":83917516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215228,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712960,"kb_used_data":81584,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93658880,"statfs":{"total":96636764160,"available":95906693120,"internally_reserved":0,"allocated":83542016,"data_stored":83319500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82694144,"data_stored":82694144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83042304,"data_stored":83042304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79958016,"data_stored":79958016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81551360,"data_stored":81551360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:22.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:23.486 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:23.487 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:23.510 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-10T20:29:22.096520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163077992,"num_objects":19911,"num_object_clones":0,"num_object_copies":79644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145307,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730368,"kb_used_data":323172,"kb_used_omap":14,"kb_used_meta":2407089,"kb_avail":374756992,"statfs":{"total":386547056640,"available":383751159808,"internally_reserved":0,"allocated":330928128,"data_stored":330038064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2464859702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,5,5,17,18,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19128320,"num_objects":-2335,"num_object_clones":0,"num_object_copies":-9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2335,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005406"},"pg_stats":[{"pgid":"1.f","version":"42'10923","reported_seq":11210,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.474954+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:19.474954+0000","last_peered":"2024-09-10T20:29:19.474954+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:19.474954+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9822208,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10923,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'10980","reported_seq":11117,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.824681+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:17.824681+0000","last_peered":"2024-09-10T20:29:17.824681+0000","last_clean":"2024-09-10T20:29:17.824681+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:17.824681+0000","last_undegraded":"2024-09-10T20:29:17.824681+0000","last_fullsized":"2024-09-10T20:29:17.824681+0000","mapping_epoch":35,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10922","reported_seq":11130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.067576+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:21.067576+0000","last_peered":"2024-09-10T20:29:21.067576+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:21.067576+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9976424,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10921,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11071","reported_seq":15244,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.798189+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:17.798189+0000","last_peered":"2024-09-10T20:29:17.798189+0000","last_clean":"2024-09-10T20:29:17.798189+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:17.798189+0000","last_undegraded":"2024-09-10T20:29:17.798189+0000","last_fullsized":"2024-09-10T20:29:17.798189+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10936320,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10902","reported_seq":11227,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.490630+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:19.490630+0000","last_peered":"2024-09-10T20:29:19.490630+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:19.490630+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10376352,"num_objects":1267,"num_object_clones":0,"num_object_copies":5068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-588}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10904","reported_seq":11131,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.492376+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:19.492376+0000","last_peered":"2024-09-10T20:29:19.492376+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:19.492376+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9781248,"num_objects":1194,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-663}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10831","reported_seq":11038,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:18.046590+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:18.046590+0000","last_peered":"2024-09-10T20:29:18.046590+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:18.046590+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10146976,"num_objects":1239,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'10989","reported_seq":11160,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:18.170859+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:18.170859+0000","last_peered":"2024-09-10T20:29:18.170859+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:18.170859+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679456,"num_objects":1304,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10773","reported_seq":14750,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:19.474706+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:19.474706+0000","last_peered":"2024-09-10T20:29:19.474706+0000","last_clean":"2024-09-10T20:29:19.474706+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:19.474706+0000","last_undegraded":"2024-09-10T20:29:19.474706+0000","last_fullsized":"2024-09-10T20:29:19.474706+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10058320,"num_objects":1228,"num_object_clones":0,"num_object_copies":4912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10915","reported_seq":14891,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:21.298875+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:21.298875+0000","last_peered":"2024-09-10T20:29:21.298875+0000","last_clean":"2024-09-10T20:29:21.298875+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:21.298875+0000","last_undegraded":"2024-09-10T20:29:21.298875+0000","last_fullsized":"2024-09-10T20:29:21.298875+0000","mapping_epoch":39,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10051584,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10855","reported_seq":11133,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.506932+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:19.506932+0000","last_peered":"2024-09-10T20:29:19.506932+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:19.506932+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10301168,"num_objects":1258,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10895","reported_seq":11198,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.547538+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:19.547538+0000","last_peered":"2024-09-10T20:29:19.547538+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:19.547538+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10402384,"num_objects":1270,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10664","reported_seq":10852,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.240165+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:21.240165+0000","last_peered":"2024-09-10T20:29:21.240165+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:21.240165+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9558608,"num_objects":1167,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10875","reported_seq":14569,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.540527+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:19.540527+0000","last_peered":"2024-09-10T20:29:19.540527+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:19.540527+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983136,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1812,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11219","reported_seq":11429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.193603+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:21.193603+0000","last_peered":"2024-09-10T20:29:21.193603+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:21.193603+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073248,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11217,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10840","reported_seq":14940,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:17.544525+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:17.544525+0000","last_peered":"2024-09-10T20:29:17.544525+0000","last_clean":"2024-09-10T20:29:17.544525+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:17.544525+0000","last_undegraded":"2024-09-10T20:29:17.544525+0000","last_fullsized":"2024-09-10T20:29:17.544525+0000","mapping_epoch":41,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10510336,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":163077992,"num_objects":19911,"num_object_clones":0,"num_object_copies":79644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145307,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327245824,"data_stored":327245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711976,"kb_used_data":80536,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93659864,"statfs":{"total":96636764160,"available":95907700736,"internally_reserved":0,"allocated":82468864,"data_stored":82246348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714000,"kb_used_data":78884,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93657840,"statfs":{"total":96636764160,"available":95905628160,"internally_reserved":0,"allocated":80777216,"data_stored":80554700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591432,"kb_used_data":82168,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93780408,"statfs":{"total":96636764160,"available":96031137792,"internally_reserved":0,"allocated":84140032,"data_stored":83917516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215228,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712960,"kb_used_data":81584,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93658880,"statfs":{"total":96636764160,"available":95906693120,"internally_reserved":0,"allocated":83542016,"data_stored":83319500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82694144,"data_stored":82694144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83042304,"data_stored":83042304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79958016,"data_stored":79958016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81551360,"data_stored":81551360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:23.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:23.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10923", 'reported_seq': 11210, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:19.474954+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:19.474954+0000', 'last_peered': '2024-09-10T20:29:19.474954+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:19.474954+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9822208, 'num_objects': 1199, 'num_object_clones': 0, 'num_object_copies': 4796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10923, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -656}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:23.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:23.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10922", 'reported_seq': 11130, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:21.067576+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:21.067576+0000', 'last_peered': '2024-09-10T20:29:21.067576+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:21.067576+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9976424, 'num_objects': 1219, 'num_object_clones': 0, 'num_object_copies': 4876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1219, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10921, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -689}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:23.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:23.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10902", 'reported_seq': 11227, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:19.490630+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:19.490630+0000', 'last_peered': '2024-09-10T20:29:19.490630+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:19.490630+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10376352, 'num_objects': 1267, 'num_object_clones': 0, 'num_object_copies': 5068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10900, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -588}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:23.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:23.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10904", 'reported_seq': 11131, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:19.492376+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:19.492376+0000', 'last_peered': '2024-09-10T20:29:19.492376+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:19.492376+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9781248, 'num_objects': 1194, 'num_object_clones': 0, 'num_object_copies': 4776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10904, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -663}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10831", 'reported_seq': 11038, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:18.046590+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:29:18.046590+0000', 'last_peered': '2024-09-10T20:29:18.046590+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:18.046590+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10146976, 'num_objects': 1239, 'num_object_clones': 0, 'num_object_copies': 4956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10829, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -654}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'10989", 'reported_seq': 11160, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:18.170859+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:18.170859+0000', 'last_peered': '2024-09-10T20:29:18.170859+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:18.170859+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10679456, 'num_objects': 1304, 'num_object_clones': 0, 'num_object_copies': 5216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10987, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10855", 'reported_seq': 11133, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:19.506932+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:19.506932+0000', 'last_peered': '2024-09-10T20:29:19.506932+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:19.506932+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10301168, 'num_objects': 1258, 'num_object_clones': 0, 'num_object_copies': 5032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10852, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -598}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10895", 'reported_seq': 11198, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:19.547538+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:19.547538+0000', 'last_peered': '2024-09-10T20:29:19.547538+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:19.547538+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10402384, 'num_objects': 1270, 'num_object_clones': 0, 'num_object_copies': 5080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10894, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:23.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:23.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10664", 'reported_seq': 10852, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:21.240165+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:21.240165+0000', 'last_peered': '2024-09-10T20:29:21.240165+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:21.240165+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9558608, 'num_objects': 1167, 'num_object_clones': 0, 'num_object_copies': 4668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10663, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -736}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:23.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:29:23.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10875", 'reported_seq': 14569, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:19.540527+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:29:19.540527+0000', 'last_peered': '2024-09-10T20:29:19.540527+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:29:19.540527+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9983136, 'num_objects': 1219, 'num_object_clones': 0, 'num_object_copies': 4876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 167, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10873, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1812, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:23.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:23.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'11219", 'reported_seq': 11429, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:21.193603+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:21.193603+0000', 'last_peered': '2024-09-10T20:29:21.193603+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:21.193603+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10073248, 'num_objects': 1230, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11217, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -678}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:23.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:24.274 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:24.275 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19076/78284 objects degraded (24.368%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":653,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":19571,"data_bytes":160292712,"bytes_used":2795634688,"bytes_avail":383751421952,"bytes_total":386547056640,"degraded_objects":19076,"degraded_total":78284,"degraded_ratio":0.24367686883654385,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:24.302 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:24.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:24.882 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:24.883 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:24.916 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-10T20:29:24.097172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":160292712,"num_objects":19571,"num_object_clones":0,"num_object_copies":78284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145647,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160598,"ondisk_log_size":160598,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730112,"kb_used_data":314588,"kb_used_omap":14,"kb_used_meta":2415409,"kb_avail":374757248,"statfs":{"total":386547056640,"available":383751421952,"internally_reserved":0,"allocated":322138112,"data_stored":321248048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2473379382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,5,4,15,11,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19570688,"num_objects":-2389,"num_object_clones":0,"num_object_copies":-9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2389,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005216"},"pg_stats":[{"pgid":"1.f","version":"42'10923","reported_seq":11210,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.474954+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:19.474954+0000","last_peered":"2024-09-10T20:29:19.474954+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:19.474954+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9822208,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10923,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'11049","reported_seq":11186,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.681751+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:22.681751+0000","last_peered":"2024-09-10T20:29:22.681751+0000","last_clean":"2024-09-10T20:29:22.681751+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:22.681751+0000","last_undegraded":"2024-09-10T20:29:22.681751+0000","last_fullsized":"2024-09-10T20:29:22.681751+0000","mapping_epoch":35,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9854976,"num_objects":1203,"num_object_clones":0,"num_object_copies":4812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10922","reported_seq":11130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.067576+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:21.067576+0000","last_peered":"2024-09-10T20:29:21.067576+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:21.067576+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9976424,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10921,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11143","reported_seq":15318,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.719809+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:22.719809+0000","last_peered":"2024-09-10T20:29:22.719809+0000","last_clean":"2024-09-10T20:29:22.719809+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:22.719809+0000","last_undegraded":"2024-09-10T20:29:22.719809+0000","last_fullsized":"2024-09-10T20:29:22.719809+0000","mapping_epoch":37,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10346496,"num_objects":1263,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10902","reported_seq":11227,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.490630+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:19.490630+0000","last_peered":"2024-09-10T20:29:19.490630+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:19.490630+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10376352,"num_objects":1267,"num_object_clones":0,"num_object_copies":5068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-588}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10904","reported_seq":11131,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.492376+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:19.492376+0000","last_peered":"2024-09-10T20:29:19.492376+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:19.492376+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9781248,"num_objects":1194,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-663}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10904","reported_seq":11112,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:23.166667+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:23.166667+0000","last_peered":"2024-09-10T20:29:23.166667+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:23.166667+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9548960,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-727}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'11052","reported_seq":11225,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:22.945857+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:22.945857+0000","last_peered":"2024-09-10T20:29:22.945857+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:22.945857+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163360,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10773","reported_seq":14750,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:19.474706+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:19.474706+0000","last_peered":"2024-09-10T20:29:19.474706+0000","last_clean":"2024-09-10T20:29:19.474706+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:19.474706+0000","last_undegraded":"2024-09-10T20:29:19.474706+0000","last_fullsized":"2024-09-10T20:29:19.474706+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10058320,"num_objects":1228,"num_object_clones":0,"num_object_copies":4912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10915","reported_seq":14891,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:21.298875+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:21.298875+0000","last_peered":"2024-09-10T20:29:21.298875+0000","last_clean":"2024-09-10T20:29:21.298875+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:21.298875+0000","last_undegraded":"2024-09-10T20:29:21.298875+0000","last_fullsized":"2024-09-10T20:29:21.298875+0000","mapping_epoch":39,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10051584,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10855","reported_seq":11133,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.506932+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:19.506932+0000","last_peered":"2024-09-10T20:29:19.506932+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:19.506932+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10301168,"num_objects":1258,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10895","reported_seq":11198,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.547538+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:19.547538+0000","last_peered":"2024-09-10T20:29:19.547538+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:19.547538+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10402384,"num_objects":1270,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10664","reported_seq":10852,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.240165+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:21.240165+0000","last_peered":"2024-09-10T20:29:21.240165+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:21.240165+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9558608,"num_objects":1167,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10875","reported_seq":14569,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:19.540527+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:19.540527+0000","last_peered":"2024-09-10T20:29:19.540527+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:19.540527+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983136,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1812,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11219","reported_seq":11429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:21.193603+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:21.193603+0000","last_peered":"2024-09-10T20:29:21.193603+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:21.193603+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073248,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11217,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10903","reported_seq":15001,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.847406+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:22.847406+0000","last_peered":"2024-09-10T20:29:22.847406+0000","last_clean":"2024-09-10T20:29:22.847406+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:22.847406+0000","last_undegraded":"2024-09-10T20:29:22.847406+0000","last_fullsized":"2024-09-10T20:29:22.847406+0000","mapping_epoch":41,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9994240,"num_objects":1220,"num_object_clones":0,"num_object_copies":4880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1776,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":160292712,"num_objects":19571,"num_object_clones":0,"num_object_copies":78284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145647,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":318681088,"data_stored":318681088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160598,"ondisk_log_size":160598,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711976,"kb_used_data":80536,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93659864,"statfs":{"total":96636764160,"available":95907700736,"internally_reserved":0,"allocated":82468864,"data_stored":82246348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714000,"kb_used_data":78884,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93657840,"statfs":{"total":96636764160,"available":95905628160,"internally_reserved":0,"allocated":80777216,"data_stored":80554700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591316,"kb_used_data":77892,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93780524,"statfs":{"total":96636764160,"available":96031256576,"internally_reserved":0,"allocated":79761408,"data_stored":79538892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525725394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,3,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712820,"kb_used_data":77276,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93659020,"statfs":{"total":96636764160,"available":95906836480,"internally_reserved":0,"allocated":79130624,"data_stored":78908108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650768079},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,5,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78434304,"data_stored":78434304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78737408,"data_stored":78737408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79958016,"data_stored":79958016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81551360,"data_stored":81551360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:27.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:28.354 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:28.354 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:28.380 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-10T20:29:28.098637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154738536,"num_objects":18893,"num_object_clones":0,"num_object_copies":75572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146325,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713620,"kb_used_data":306420,"kb_used_omap":14,"kb_used_meta":2407089,"kb_avail":374773740,"statfs":{"total":386547056640,"available":383768309760,"internally_reserved":0,"allocated":313774080,"data_stored":312884016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2464859702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,5,22,10,8,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18948096,"num_objects":-2313,"num_object_clones":0,"num_object_copies":-9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005020"},"pg_stats":[{"pgid":"1.f","version":"42'10987","reported_seq":11273,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.439168+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:24.439168+0000","last_peered":"2024-09-10T20:29:24.439168+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:24.439168+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9297920,"num_objects":1135,"num_object_clones":0,"num_object_copies":4540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'11049","reported_seq":11186,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.681751+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:22.681751+0000","last_peered":"2024-09-10T20:29:22.681751+0000","last_clean":"2024-09-10T20:29:22.681751+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:22.681751+0000","last_undegraded":"2024-09-10T20:29:22.681751+0000","last_fullsized":"2024-09-10T20:29:22.681751+0000","mapping_epoch":35,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9854976,"num_objects":1203,"num_object_clones":0,"num_object_copies":4812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10984","reported_seq":11192,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.274813+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:26.274813+0000","last_peered":"2024-09-10T20:29:26.274813+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:26.274813+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9468520,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10983,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11143","reported_seq":15318,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.719809+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:22.719809+0000","last_peered":"2024-09-10T20:29:22.719809+0000","last_clean":"2024-09-10T20:29:22.719809+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:22.719809+0000","last_undegraded":"2024-09-10T20:29:22.719809+0000","last_fullsized":"2024-09-10T20:29:22.719809+0000","mapping_epoch":37,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10346496,"num_objects":1263,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10958","reported_seq":11283,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.358664+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:24.358664+0000","last_peered":"2024-09-10T20:29:24.358664+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:24.358664+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9917600,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10956,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10958","reported_seq":11183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.485586+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:24.485586+0000","last_peered":"2024-09-10T20:29:24.485586+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:24.485586+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9338880,"num_objects":1140,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10958,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10904","reported_seq":11112,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:23.166667+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:23.166667+0000","last_peered":"2024-09-10T20:29:23.166667+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:23.166667+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9548960,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-727}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'11052","reported_seq":11225,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:22.945857+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:22.945857+0000","last_peered":"2024-09-10T20:29:22.945857+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:22.945857+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163360,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10844","reported_seq":14821,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:24.604062+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:24.604062+0000","last_peered":"2024-09-10T20:29:24.604062+0000","last_clean":"2024-09-10T20:29:24.604062+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:24.604062+0000","last_undegraded":"2024-09-10T20:29:24.604062+0000","last_fullsized":"2024-09-10T20:29:24.604062+0000","mapping_epoch":33,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9476688,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7397,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10969","reported_seq":14945,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:26.224747+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:26.224747+0000","last_peered":"2024-09-10T20:29:26.224747+0000","last_clean":"2024-09-10T20:29:26.224747+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:26.224747+0000","last_undegraded":"2024-09-10T20:29:26.224747+0000","last_fullsized":"2024-09-10T20:29:26.224747+0000","mapping_epoch":39,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9609216,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10915","reported_seq":11193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.201608+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:24.201608+0000","last_peered":"2024-09-10T20:29:24.201608+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:24.201608+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9809648,"num_objects":1198,"num_object_clones":0,"num_object_copies":4792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10972","reported_seq":11275,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.517746+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:24.517746+0000","last_peered":"2024-09-10T20:29:24.517746+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:24.517746+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9771600,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-667}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10723","reported_seq":10911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.380073+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:26.380073+0000","last_peered":"2024-09-10T20:29:26.380073+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:26.380073+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075280,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10937","reported_seq":14744,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.626596+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:24.626596+0000","last_peered":"2024-09-10T20:29:24.626596+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:24.626596+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9475232,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1046}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11278","reported_seq":11486,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.467155+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:26.467155+0000","last_peered":"2024-09-10T20:29:26.467155+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:26.467155+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9589920,"num_objects":1171,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10903","reported_seq":15001,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.847406+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:22.847406+0000","last_peered":"2024-09-10T20:29:22.847406+0000","last_clean":"2024-09-10T20:29:22.847406+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:22.847406+0000","last_undegraded":"2024-09-10T20:29:22.847406+0000","last_fullsized":"2024-09-10T20:29:22.847406+0000","mapping_epoch":41,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9994240,"num_objects":1220,"num_object_clones":0,"num_object_copies":4880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1776,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154738536,"num_objects":18893,"num_object_clones":0,"num_object_copies":75572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146325,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310292480,"data_stored":310292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711992,"kb_used_data":76372,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93659848,"statfs":{"total":96636764160,"available":95907684352,"internally_reserved":0,"allocated":78204928,"data_stored":77982412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,14,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697492,"kb_used_data":74880,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93674348,"statfs":{"total":96636764160,"available":95922532352,"internally_reserved":0,"allocated":76677120,"data_stored":76454604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,4,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591316,"kb_used_data":77892,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93780524,"statfs":{"total":96636764160,"available":96031256576,"internally_reserved":0,"allocated":79761408,"data_stored":79538892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525725394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,3,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712820,"kb_used_data":77276,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93659020,"statfs":{"total":96636764160,"available":95906836480,"internally_reserved":0,"allocated":79130624,"data_stored":78908108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650768079},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,5,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78434304,"data_stored":78434304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78737408,"data_stored":78737408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75849728,"data_stored":75849728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77271040,"data_stored":77271040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:28.381 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:28.987 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:29.005 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-10T20:29:28.098637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154738536,"num_objects":18893,"num_object_clones":0,"num_object_copies":75572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146325,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713620,"kb_used_data":306420,"kb_used_omap":14,"kb_used_meta":2407089,"kb_avail":374773740,"statfs":{"total":386547056640,"available":383768309760,"internally_reserved":0,"allocated":313774080,"data_stored":312884016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2464859702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,5,22,10,8,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18948096,"num_objects":-2313,"num_object_clones":0,"num_object_copies":-9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005020"},"pg_stats":[{"pgid":"1.f","version":"42'10987","reported_seq":11273,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.439168+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:24.439168+0000","last_peered":"2024-09-10T20:29:24.439168+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:24.439168+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9297920,"num_objects":1135,"num_object_clones":0,"num_object_copies":4540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'11049","reported_seq":11186,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.681751+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:22.681751+0000","last_peered":"2024-09-10T20:29:22.681751+0000","last_clean":"2024-09-10T20:29:22.681751+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:22.681751+0000","last_undegraded":"2024-09-10T20:29:22.681751+0000","last_fullsized":"2024-09-10T20:29:22.681751+0000","mapping_epoch":35,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9854976,"num_objects":1203,"num_object_clones":0,"num_object_copies":4812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10984","reported_seq":11192,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.274813+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:26.274813+0000","last_peered":"2024-09-10T20:29:26.274813+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:26.274813+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9468520,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10983,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11143","reported_seq":15318,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.719809+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:22.719809+0000","last_peered":"2024-09-10T20:29:22.719809+0000","last_clean":"2024-09-10T20:29:22.719809+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:22.719809+0000","last_undegraded":"2024-09-10T20:29:22.719809+0000","last_fullsized":"2024-09-10T20:29:22.719809+0000","mapping_epoch":37,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10346496,"num_objects":1263,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'10958","reported_seq":11283,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.358664+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:24.358664+0000","last_peered":"2024-09-10T20:29:24.358664+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:24.358664+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9917600,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10956,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10958","reported_seq":11183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.485586+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:24.485586+0000","last_peered":"2024-09-10T20:29:24.485586+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:24.485586+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9338880,"num_objects":1140,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10958,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10904","reported_seq":11112,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:23.166667+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:23.166667+0000","last_peered":"2024-09-10T20:29:23.166667+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:23.166667+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9548960,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-727}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'11052","reported_seq":11225,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:22.945857+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:22.945857+0000","last_peered":"2024-09-10T20:29:22.945857+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:22.945857+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163360,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10844","reported_seq":14821,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:24.604062+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:24.604062+0000","last_peered":"2024-09-10T20:29:24.604062+0000","last_clean":"2024-09-10T20:29:24.604062+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:24.604062+0000","last_undegraded":"2024-09-10T20:29:24.604062+0000","last_fullsized":"2024-09-10T20:29:24.604062+0000","mapping_epoch":33,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9476688,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7397,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10969","reported_seq":14945,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:26.224747+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:26.224747+0000","last_peered":"2024-09-10T20:29:26.224747+0000","last_clean":"2024-09-10T20:29:26.224747+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:26.224747+0000","last_undegraded":"2024-09-10T20:29:26.224747+0000","last_fullsized":"2024-09-10T20:29:26.224747+0000","mapping_epoch":39,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9609216,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10915","reported_seq":11193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.201608+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:24.201608+0000","last_peered":"2024-09-10T20:29:24.201608+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:24.201608+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9809648,"num_objects":1198,"num_object_clones":0,"num_object_copies":4792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'10972","reported_seq":11275,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.517746+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:24.517746+0000","last_peered":"2024-09-10T20:29:24.517746+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:24.517746+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9771600,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-667}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10723","reported_seq":10911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.380073+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:26.380073+0000","last_peered":"2024-09-10T20:29:26.380073+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:26.380073+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075280,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10937","reported_seq":14744,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:24.626596+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:24.626596+0000","last_peered":"2024-09-10T20:29:24.626596+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:24.626596+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9475232,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1046}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11278","reported_seq":11486,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.467155+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:26.467155+0000","last_peered":"2024-09-10T20:29:26.467155+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:26.467155+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9589920,"num_objects":1171,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10903","reported_seq":15001,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:22.847406+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:22.847406+0000","last_peered":"2024-09-10T20:29:22.847406+0000","last_clean":"2024-09-10T20:29:22.847406+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:22.847406+0000","last_undegraded":"2024-09-10T20:29:22.847406+0000","last_fullsized":"2024-09-10T20:29:22.847406+0000","mapping_epoch":41,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9994240,"num_objects":1220,"num_object_clones":0,"num_object_copies":4880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1776,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154738536,"num_objects":18893,"num_object_clones":0,"num_object_copies":75572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146325,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310292480,"data_stored":310292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711992,"kb_used_data":76372,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93659848,"statfs":{"total":96636764160,"available":95907684352,"internally_reserved":0,"allocated":78204928,"data_stored":77982412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,14,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697492,"kb_used_data":74880,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93674348,"statfs":{"total":96636764160,"available":95922532352,"internally_reserved":0,"allocated":76677120,"data_stored":76454604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,4,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591316,"kb_used_data":77892,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93780524,"statfs":{"total":96636764160,"available":96031256576,"internally_reserved":0,"allocated":79761408,"data_stored":79538892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525725394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,3,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712820,"kb_used_data":77276,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93659020,"statfs":{"total":96636764160,"available":95906836480,"internally_reserved":0,"allocated":79130624,"data_stored":78908108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650768079},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,5,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78434304,"data_stored":78434304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78737408,"data_stored":78737408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75849728,"data_stored":75849728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77271040,"data_stored":77271040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:29.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:29.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'10987", 'reported_seq': 11273, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:24.439168+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:24.439168+0000', 'last_peered': '2024-09-10T20:29:24.439168+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:24.439168+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9297920, 'num_objects': 1135, 'num_object_clones': 0, 'num_object_copies': 4540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10987, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -720}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:29.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:29.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "42'10984", 'reported_seq': 11192, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:26.274813+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:26.274813+0000', 'last_peered': '2024-09-10T20:29:26.274813+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:26.274813+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9468520, 'num_objects': 1157, 'num_object_clones': 0, 'num_object_copies': 4628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1157, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10983, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -751}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'10958", 'reported_seq': 11283, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:24.358664+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:24.358664+0000', 'last_peered': '2024-09-10T20:29:24.358664+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:24.358664+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9917600, 'num_objects': 1211, 'num_object_clones': 0, 'num_object_copies': 4844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10956, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -644}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10958", 'reported_seq': 11183, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:24.485586+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:24.485586+0000', 'last_peered': '2024-09-10T20:29:24.485586+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:24.485586+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9338880, 'num_objects': 1140, 'num_object_clones': 0, 'num_object_copies': 4560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10958, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -717}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "42'10904", 'reported_seq': 11112, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:23.166667+0000', 'last_change': '2024-09-10T20:20:16.155671+0000', 'last_active': '2024-09-10T20:29:23.166667+0000', 'last_peered': '2024-09-10T20:29:23.166667+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:23.166667+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9548960, 'num_objects': 1166, 'num_object_clones': 0, 'num_object_copies': 4664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1895, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10902, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -727}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "42'11052", 'reported_seq': 11225, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:22.945857+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:22.945857+0000', 'last_peered': '2024-09-10T20:29:22.945857+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:22.945857+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10163360, 'num_objects': 1241, 'num_object_clones': 0, 'num_object_copies': 4964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11050, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -759}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10915", 'reported_seq': 11193, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:24.201608+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:24.201608+0000', 'last_peered': '2024-09-10T20:29:24.201608+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:24.201608+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9809648, 'num_objects': 1198, 'num_object_clones': 0, 'num_object_copies': 4792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10912, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -658}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:29.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'10972", 'reported_seq': 11275, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:24.517746+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:24.517746+0000', 'last_peered': '2024-09-10T20:29:24.517746+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:24.517746+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9771600, 'num_objects': 1193, 'num_object_clones': 0, 'num_object_copies': 4772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10971, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:29.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:29.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10723", 'reported_seq': 10911, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:26.380073+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:26.380073+0000', 'last_peered': '2024-09-10T20:29:26.380073+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:26.380073+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9075280, 'num_objects': 1108, 'num_object_clones': 0, 'num_object_copies': 4432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10722, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:29.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:29:29.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "42'10937", 'reported_seq': 14744, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:24.626596+0000', 'last_change': '2024-09-10T20:26:59.275085+0000', 'last_active': '2024-09-10T20:29:24.626596+0000', 'last_peered': '2024-09-10T20:29:24.626596+0000', 'last_clean': '2024-09-10T20:19:05.724464+0000', 'last_became_active': '2024-09-10T20:20:16.094222+0000', 'last_became_peered': '2024-09-10T20:20:16.094222+0000', 'last_unstale': '2024-09-10T20:29:24.626596+0000', 'last_undegraded': '2024-09-10T20:20:15.515058+0000', 'last_fullsized': '2024-09-10T20:20:15.455156+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9475232, 'num_objects': 1157, 'num_object_clones': 0, 'num_object_copies': 4628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 111, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10935, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1868, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1046}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:29.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:29.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "42'11278", 'reported_seq': 11486, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:26.467155+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:26.467155+0000', 'last_peered': '2024-09-10T20:29:26.467155+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:26.467155+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9589920, 'num_objects': 1171, 'num_object_clones': 0, 'num_object_copies': 4684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11276, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:29.008 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:29.803 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:29.823 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19020/75572 objects degraded (25.168%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":658,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18893,"data_bytes":154738536,"bytes_used":2778746880,"bytes_avail":383768309760,"bytes_total":386547056640,"degraded_objects":19020,"degraded_total":75572,"degraded_ratio":0.25168051659344731,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":192},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:29.825 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:29.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:30.476 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:30.476 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:30.516 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-10T20:29:30.099318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148995944,"num_objects":18192,"num_object_clones":0,"num_object_copies":72768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147026,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2776,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680740,"kb_used_data":294352,"kb_used_omap":14,"kb_used_meta":2386289,"kb_avail":374806620,"statfs":{"total":386547056640,"available":383801978880,"internally_reserved":0,"allocated":301416448,"data_stored":300526384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2443560502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,3,16,13,8,16,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20897792,"num_objects":-2551,"num_object_clones":0,"num_object_copies":-10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2551,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004892"},"pg_stats":[{"pgid":"1.f","version":"42'11037","reported_seq":11324,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.667462+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:29.667462+0000","last_peered":"2024-09-10T20:29:29.667462+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:29.667462+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8888320,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'11115","reported_seq":11252,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:27.969380+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:27.969380+0000","last_peered":"2024-09-10T20:29:27.969380+0000","last_clean":"2024-09-10T20:29:27.969380+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:27.969380+0000","last_undegraded":"2024-09-10T20:29:27.969380+0000","last_fullsized":"2024-09-10T20:29:27.969380+0000","mapping_epoch":35,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9314304,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7556,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'10984","reported_seq":11192,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.274813+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:26.274813+0000","last_peered":"2024-09-10T20:29:26.274813+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:26.274813+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9468520,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10983,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"42'11211","reported_seq":15385,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:28.016708+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:28.016708+0000","last_peered":"2024-09-10T20:29:28.016708+0000","last_clean":"2024-09-10T20:29:28.016708+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:28.016708+0000","last_undegraded":"2024-09-10T20:29:28.016708+0000","last_fullsized":"2024-09-10T20:29:28.016708+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9789440,"num_objects":1195,"num_object_clones":0,"num_object_copies":4780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5471,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'11025","reported_seq":11350,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.703454+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:29.703454+0000","last_peered":"2024-09-10T20:29:29.703454+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:29.703454+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368736,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10992","reported_seq":11219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.186613+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:29.186613+0000","last_peered":"2024-09-10T20:29:29.186613+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:29.186613+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9060352,"num_objects":1106,"num_object_clones":0,"num_object_copies":4424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"42'10951","reported_seq":11159,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:27.709434+0000","last_change":"2024-09-10T20:20:16.155671+0000","last_active":"2024-09-10T20:29:27.709434+0000","last_peered":"2024-09-10T20:29:27.709434+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:27.709434+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9163936,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-774}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'11117","reported_seq":11290,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:27.989128+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:27.989128+0000","last_peered":"2024-09-10T20:29:27.989128+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:27.989128+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9630880,"num_objects":1176,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11115,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10903","reported_seq":14879,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:29.729612+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:29.729612+0000","last_peered":"2024-09-10T20:29:29.729612+0000","last_clean":"2024-09-10T20:29:29.729612+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:29.729612+0000","last_undegraded":"2024-09-10T20:29:29.729612+0000","last_fullsized":"2024-09-10T20:29:29.729612+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8993360,"num_objects":1098,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'10969","reported_seq":14945,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:26.224747+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:26.224747+0000","last_peered":"2024-09-10T20:29:26.224747+0000","last_clean":"2024-09-10T20:29:26.224747+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:26.224747+0000","last_undegraded":"2024-09-10T20:29:26.224747+0000","last_fullsized":"2024-09-10T20:29:26.224747+0000","mapping_epoch":39,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9609216,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10991","reported_seq":11269,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.680805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:29.680805+0000","last_peered":"2024-09-10T20:29:29.680805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:29.680805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9187056,"num_objects":1122,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'11022","reported_seq":11325,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.422021+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:29.422021+0000","last_peered":"2024-09-10T20:29:29.422021+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:29.422021+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362000,"num_objects":1143,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11021,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10723","reported_seq":10911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.380073+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:26.380073+0000","last_peered":"2024-09-10T20:29:26.380073+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:26.380073+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075280,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"42'10992","reported_seq":15005,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.728382+0000","last_change":"2024-09-10T20:26:59.275085+0000","last_active":"2024-09-10T20:29:29.728382+0000","last_peered":"2024-09-10T20:29:29.728382+0000","last_clean":"2024-09-10T20:19:05.724464+0000","last_became_active":"2024-09-10T20:20:16.094222+0000","last_became_peered":"2024-09-10T20:20:16.094222+0000","last_unstale":"2024-09-10T20:29:29.728382+0000","last_undegraded":"2024-09-10T20:20:15.515058+0000","last_fullsized":"2024-09-10T20:20:15.455156+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9024672,"num_objects":1102,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10990,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"42'11278","reported_seq":11486,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:26.467155+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:26.467155+0000","last_peered":"2024-09-10T20:29:26.467155+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:26.467155+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9589920,"num_objects":1171,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"42'10967","reported_seq":15068,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:27.849563+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:27.849563+0000","last_peered":"2024-09-10T20:29:27.849563+0000","last_clean":"2024-09-10T20:29:27.849563+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:27.849563+0000","last_undegraded":"2024-09-10T20:29:27.849563+0000","last_fullsized":"2024-09-10T20:29:27.849563+0000","mapping_epoch":41,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9469952,"num_objects":1156,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1840,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148995944,"num_objects":18192,"num_object_clones":0,"num_object_copies":72768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147026,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2776,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298426368,"data_stored":298426368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712424,"kb_used_data":72648,"kb_used_omap":4,"kb_used_meta":639739,"kb_avail":93659416,"statfs":{"total":96636764160,"available":95907241984,"internally_reserved":0,"allocated":74391552,"data_stored":74169036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":655093456},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697492,"kb_used_data":74880,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93674348,"statfs":{"total":96636764160,"available":95922532352,"internally_reserved":0,"allocated":76677120,"data_stored":76454604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,4,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574372,"kb_used_data":73428,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93797468,"statfs":{"total":96636764160,"available":96048607232,"internally_reserved":0,"allocated":75190272,"data_stored":74967756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512945874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,6,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215230,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696452,"kb_used_data":73396,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93675388,"statfs":{"total":96636764160,"available":95923597312,"internally_reserved":0,"allocated":75157504,"data_stored":74934988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74534912,"data_stored":74534912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74653696,"data_stored":74653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75849728,"data_stored":75849728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73388032,"data_stored":73388032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:33.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:34.113 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:34.113 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:34.147 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-10T20:29:34.101175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143794024,"num_objects":17557,"num_object_clones":0,"num_object_copies":70228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147661,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718100,"kb_used_data":283112,"kb_used_omap":14,"kb_used_meta":2434865,"kb_avail":374769260,"statfs":{"total":386547056640,"available":383763722240,"internally_reserved":0,"allocated":289906688,"data_stored":288997566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2493302326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,10,14,12,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12861440,"num_objects":-1570,"num_object_clones":0,"num_object_copies":-6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003497"},"pg_stats":[{"pgid":"1.f","version":"42'11037","reported_seq":11324,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.667462+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:29.667462+0000","last_peered":"2024-09-10T20:29:29.667462+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:29.667462+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8888320,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11183","reported_seq":11324,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.092566+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:33.092566+0000","last_peered":"2024-09-10T20:29:33.092566+0000","last_clean":"2024-09-10T20:29:33.092566+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:33.092566+0000","last_undegraded":"2024-09-10T20:29:33.092566+0000","last_fullsized":"2024-09-10T20:29:33.092566+0000","mapping_epoch":35,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757248,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"43'11062","reported_seq":11271,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.430539+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:31.430539+0000","last_peered":"2024-09-10T20:29:31.430539+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:31.430539+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8829544,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11061,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11277","reported_seq":15456,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.219887+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:33.219887+0000","last_peered":"2024-09-10T20:29:33.219887+0000","last_clean":"2024-09-10T20:29:33.219887+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:33.219887+0000","last_undegraded":"2024-09-10T20:29:33.219887+0000","last_fullsized":"2024-09-10T20:29:33.219887+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9248768,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'11025","reported_seq":11350,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.703454+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:29.703454+0000","last_peered":"2024-09-10T20:29:29.703454+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:29.703454+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368736,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10992","reported_seq":11219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.186613+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:29.186613+0000","last_peered":"2024-09-10T20:29:29.186613+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:29.186613+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9060352,"num_objects":1106,"num_object_clones":0,"num_object_copies":4424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11020","reported_seq":11355,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.217643+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:33.217643+0000","last_peered":"2024-09-10T20:29:33.217643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:33.217643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598688,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-783}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11194","reported_seq":11371,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.143713+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:33.143713+0000","last_peered":"2024-09-10T20:29:33.143713+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:33.143713+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9000096,"num_objects":1099,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-901}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10903","reported_seq":14879,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:29.729612+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:29.729612+0000","last_peered":"2024-09-10T20:29:29.729612+0000","last_clean":"2024-09-10T20:29:29.729612+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:29.729612+0000","last_undegraded":"2024-09-10T20:29:29.729612+0000","last_fullsized":"2024-09-10T20:29:29.729612+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8993360,"num_objects":1098,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"43'11030","reported_seq":15008,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:29:31.440335+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:31.440335+0000","last_peered":"2024-09-10T20:29:31.440335+0000","last_clean":"2024-09-10T20:29:31.440335+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:31.440335+0000","last_undegraded":"2024-09-10T20:29:31.440335+0000","last_fullsized":"2024-09-10T20:29:31.440335+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109504,"num_objects":1112,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10991","reported_seq":11269,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.680805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:29.680805+0000","last_peered":"2024-09-10T20:29:29.680805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:29.680805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9187056,"num_objects":1122,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'11022","reported_seq":11325,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.422021+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:29.422021+0000","last_peered":"2024-09-10T20:29:29.422021+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:29.422021+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362000,"num_objects":1143,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11021,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10777","reported_seq":10967,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.262565+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:31.262565+0000","last_peered":"2024-09-10T20:29:31.262565+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:31.262565+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8632912,"num_objects":1054,"num_object_clones":0,"num_object_copies":4216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-849}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11041","reported_seq":15085,"reported_epoch":44,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:29:33.216099+0000","last_change":"2024-09-10T20:29:33.035436+0000","last_active":"2024-09-10T20:29:33.216099+0000","last_peered":"2024-09-10T20:29:33.216099+0000","last_clean":"2024-09-10T20:29:33.216099+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:33.216099+0000","last_undegraded":"2024-09-10T20:29:33.216099+0000","last_fullsized":"2024-09-10T20:29:33.216099+0000","mapping_epoch":43,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":99,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623264,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"43'11331","reported_seq":11543,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.377106+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:31.377106+0000","last_peered":"2024-09-10T20:29:31.377106+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:31.377106+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9155744,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11329,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11027","reported_seq":15132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.163591+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:33.163591+0000","last_peered":"2024-09-10T20:29:33.163591+0000","last_clean":"2024-09-10T20:29:33.163591+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:33.163591+0000","last_undegraded":"2024-09-10T20:29:33.163591+0000","last_fullsized":"2024-09-10T20:29:33.163591+0000","mapping_epoch":41,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":1096,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":143794024,"num_objects":17557,"num_object_clones":0,"num_object_copies":70228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147661,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286605312,"data_stored":286605312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712424,"kb_used_data":72648,"kb_used_omap":4,"kb_used_meta":639739,"kb_avail":93659416,"statfs":{"total":96636764160,"available":95907241984,"internally_reserved":0,"allocated":74391552,"data_stored":74169036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":655093456},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715840,"kb_used_data":71020,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93656000,"statfs":{"total":96636764160,"available":95903744000,"internally_reserved":0,"allocated":72724480,"data_stored":72501964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574700,"kb_used_data":69584,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93797140,"statfs":{"total":96636764160,"available":96048271360,"internally_reserved":0,"allocated":71254016,"data_stored":71021971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715136,"kb_used_data":69860,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93656704,"statfs":{"total":96636764160,"available":95904464896,"internally_reserved":0,"allocated":71536640,"data_stored":71304595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70627328,"data_stored":70627328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70668288,"data_stored":70668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71921664,"data_stored":71921664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73388032,"data_stored":73388032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:34.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:34.741 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:34.742 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:34.765 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-10T20:29:34.101175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143794024,"num_objects":17557,"num_object_clones":0,"num_object_copies":70228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147661,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718100,"kb_used_data":283112,"kb_used_omap":14,"kb_used_meta":2434865,"kb_avail":374769260,"statfs":{"total":386547056640,"available":383763722240,"internally_reserved":0,"allocated":289906688,"data_stored":288997566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2493302326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,10,14,12,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12861440,"num_objects":-1570,"num_object_clones":0,"num_object_copies":-6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003497"},"pg_stats":[{"pgid":"1.f","version":"42'11037","reported_seq":11324,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.667462+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:29.667462+0000","last_peered":"2024-09-10T20:29:29.667462+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:29.667462+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8888320,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11183","reported_seq":11324,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.092566+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:33.092566+0000","last_peered":"2024-09-10T20:29:33.092566+0000","last_clean":"2024-09-10T20:29:33.092566+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:33.092566+0000","last_undegraded":"2024-09-10T20:29:33.092566+0000","last_fullsized":"2024-09-10T20:29:33.092566+0000","mapping_epoch":35,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757248,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"43'11062","reported_seq":11271,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.430539+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:31.430539+0000","last_peered":"2024-09-10T20:29:31.430539+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:31.430539+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8829544,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11061,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11277","reported_seq":15456,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.219887+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:33.219887+0000","last_peered":"2024-09-10T20:29:33.219887+0000","last_clean":"2024-09-10T20:29:33.219887+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:33.219887+0000","last_undegraded":"2024-09-10T20:29:33.219887+0000","last_fullsized":"2024-09-10T20:29:33.219887+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9248768,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'11025","reported_seq":11350,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.703454+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:29.703454+0000","last_peered":"2024-09-10T20:29:29.703454+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:29.703454+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368736,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10992","reported_seq":11219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.186613+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:29.186613+0000","last_peered":"2024-09-10T20:29:29.186613+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:29.186613+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9060352,"num_objects":1106,"num_object_clones":0,"num_object_copies":4424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11020","reported_seq":11355,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.217643+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:33.217643+0000","last_peered":"2024-09-10T20:29:33.217643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:33.217643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598688,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-783}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11194","reported_seq":11371,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.143713+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:33.143713+0000","last_peered":"2024-09-10T20:29:33.143713+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:33.143713+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9000096,"num_objects":1099,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-901}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10903","reported_seq":14879,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:29.729612+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:29.729612+0000","last_peered":"2024-09-10T20:29:29.729612+0000","last_clean":"2024-09-10T20:29:29.729612+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:29.729612+0000","last_undegraded":"2024-09-10T20:29:29.729612+0000","last_fullsized":"2024-09-10T20:29:29.729612+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8993360,"num_objects":1098,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"43'11030","reported_seq":15008,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:29:31.440335+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:31.440335+0000","last_peered":"2024-09-10T20:29:31.440335+0000","last_clean":"2024-09-10T20:29:31.440335+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:31.440335+0000","last_undegraded":"2024-09-10T20:29:31.440335+0000","last_fullsized":"2024-09-10T20:29:31.440335+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109504,"num_objects":1112,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10991","reported_seq":11269,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.680805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:29.680805+0000","last_peered":"2024-09-10T20:29:29.680805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:29.680805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9187056,"num_objects":1122,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'11022","reported_seq":11325,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.422021+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:29.422021+0000","last_peered":"2024-09-10T20:29:29.422021+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:29.422021+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362000,"num_objects":1143,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11021,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10777","reported_seq":10967,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.262565+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:31.262565+0000","last_peered":"2024-09-10T20:29:31.262565+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:31.262565+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8632912,"num_objects":1054,"num_object_clones":0,"num_object_copies":4216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-849}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11041","reported_seq":15085,"reported_epoch":44,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:29:33.216099+0000","last_change":"2024-09-10T20:29:33.035436+0000","last_active":"2024-09-10T20:29:33.216099+0000","last_peered":"2024-09-10T20:29:33.216099+0000","last_clean":"2024-09-10T20:29:33.216099+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:33.216099+0000","last_undegraded":"2024-09-10T20:29:33.216099+0000","last_fullsized":"2024-09-10T20:29:33.216099+0000","mapping_epoch":43,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":99,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623264,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"43'11331","reported_seq":11543,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.377106+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:31.377106+0000","last_peered":"2024-09-10T20:29:31.377106+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:31.377106+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9155744,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11329,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11027","reported_seq":15132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.163591+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:33.163591+0000","last_peered":"2024-09-10T20:29:33.163591+0000","last_clean":"2024-09-10T20:29:33.163591+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:33.163591+0000","last_undegraded":"2024-09-10T20:29:33.163591+0000","last_fullsized":"2024-09-10T20:29:33.163591+0000","mapping_epoch":41,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":1096,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":143794024,"num_objects":17557,"num_object_clones":0,"num_object_copies":70228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147661,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286605312,"data_stored":286605312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712424,"kb_used_data":72648,"kb_used_omap":4,"kb_used_meta":639739,"kb_avail":93659416,"statfs":{"total":96636764160,"available":95907241984,"internally_reserved":0,"allocated":74391552,"data_stored":74169036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":655093456},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715840,"kb_used_data":71020,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93656000,"statfs":{"total":96636764160,"available":95903744000,"internally_reserved":0,"allocated":72724480,"data_stored":72501964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574700,"kb_used_data":69584,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93797140,"statfs":{"total":96636764160,"available":96048271360,"internally_reserved":0,"allocated":71254016,"data_stored":71021971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715136,"kb_used_data":69860,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93656704,"statfs":{"total":96636764160,"available":95904464896,"internally_reserved":0,"allocated":71536640,"data_stored":71304595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70627328,"data_stored":70627328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70668288,"data_stored":70668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71921664,"data_stored":71921664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73388032,"data_stored":73388032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "42'11037", 'reported_seq': 11324, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:29.667462+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:29.667462+0000', 'last_peered': '2024-09-10T20:29:29.667462+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:29.667462+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8888320, 'num_objects': 1085, 'num_object_clones': 0, 'num_object_copies': 4340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11037, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "43'11062", 'reported_seq': 11271, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:31.430539+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:31.430539+0000', 'last_peered': '2024-09-10T20:29:31.430539+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:31.430539+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8829544, 'num_objects': 1079, 'num_object_clones': 0, 'num_object_copies': 4316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1079, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11061, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -829}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "42'11025", 'reported_seq': 11350, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:29.703454+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:29.703454+0000', 'last_peered': '2024-09-10T20:29:29.703454+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:29.703454+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9368736, 'num_objects': 1144, 'num_object_clones': 0, 'num_object_copies': 4576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11023, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -711}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "42'10992", 'reported_seq': 11219, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:29.186613+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:29.186613+0000', 'last_peered': '2024-09-10T20:29:29.186613+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:29.186613+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9060352, 'num_objects': 1106, 'num_object_clones': 0, 'num_object_copies': 4424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10992, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:34.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11020", 'reported_seq': 11355, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:33.217643+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:29:33.217643+0000', 'last_peered': '2024-09-10T20:29:33.217643+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:33.217643+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8598688, 'num_objects': 1050, 'num_object_clones': 0, 'num_object_copies': 4200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1833, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11018, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 124, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -783}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11194", 'reported_seq': 11371, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:33.143713+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:33.143713+0000', 'last_peered': '2024-09-10T20:29:33.143713+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:33.143713+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9000096, 'num_objects': 1099, 'num_object_clones': 0, 'num_object_copies': 4396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11192, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "42'10991", 'reported_seq': 11269, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:29.680805+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:29.680805+0000', 'last_peered': '2024-09-10T20:29:29.680805+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:29.680805+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9187056, 'num_objects': 1122, 'num_object_clones': 0, 'num_object_copies': 4488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10988, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -734}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "42'11022", 'reported_seq': 11325, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:29.422021+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:29.422021+0000', 'last_peered': '2024-09-10T20:29:29.422021+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:29.422021+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9362000, 'num_objects': 1143, 'num_object_clones': 0, 'num_object_copies': 4572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11021, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "42'10777", 'reported_seq': 10967, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:31.262565+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:31.262565+0000', 'last_peered': '2024-09-10T20:29:31.262565+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:31.262565+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8632912, 'num_objects': 1054, 'num_object_clones': 0, 'num_object_copies': 4216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10776, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "44'11041", 'reported_seq': 15085, 'reported_epoch': 44, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-10T20:29:33.216099+0000', 'last_change': '2024-09-10T20:29:33.035436+0000', 'last_active': '2024-09-10T20:29:33.216099+0000', 'last_peered': '2024-09-10T20:29:33.216099+0000', 'last_clean': '2024-09-10T20:29:33.216099+0000', 'last_became_active': '2024-09-10T20:29:32.300730+0000', 'last_became_peered': '2024-09-10T20:29:32.300730+0000', 'last_unstale': '2024-09-10T20:29:33.216099+0000', 'last_undegraded': '2024-09-10T20:29:33.216099+0000', 'last_fullsized': '2024-09-10T20:29:33.216099+0000', 'mapping_epoch': 43, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 99, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8623264, 'num_objects': 1053, 'num_object_clones': 0, 'num_object_copies': 4212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11039, 'num_write_kb': 48372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1979, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [1, 3, 2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:34.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "43'11331", 'reported_seq': 11543, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:31.377106+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:31.377106+0000', 'last_peered': '2024-09-10T20:29:31.377106+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:31.377106+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9155744, 'num_objects': 1118, 'num_object_clones': 0, 'num_object_copies': 4472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11329, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:34.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:35.481 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:35.481 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18847/70228 objects degraded (26.837%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":664,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":5},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17557,"data_bytes":143794024,"bytes_used":2783334400,"bytes_avail":383763722240,"bytes_total":386547056640,"degraded_objects":18847,"degraded_total":70228,"degraded_ratio":0.26836874181238252,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:35.510 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:35.510 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:36.011 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:36.011 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:36.034 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-10T20:29:34.101175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143794024,"num_objects":17557,"num_object_clones":0,"num_object_copies":70228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147661,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718100,"kb_used_data":283112,"kb_used_omap":14,"kb_used_meta":2434865,"kb_avail":374769260,"statfs":{"total":386547056640,"available":383763722240,"internally_reserved":0,"allocated":289906688,"data_stored":288997566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2493302326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,10,14,12,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12861440,"num_objects":-1570,"num_object_clones":0,"num_object_copies":-6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003497"},"pg_stats":[{"pgid":"1.f","version":"42'11037","reported_seq":11324,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.667462+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:29.667462+0000","last_peered":"2024-09-10T20:29:29.667462+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:29.667462+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8888320,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11183","reported_seq":11324,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.092566+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:33.092566+0000","last_peered":"2024-09-10T20:29:33.092566+0000","last_clean":"2024-09-10T20:29:33.092566+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:33.092566+0000","last_undegraded":"2024-09-10T20:29:33.092566+0000","last_fullsized":"2024-09-10T20:29:33.092566+0000","mapping_epoch":35,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757248,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"43'11062","reported_seq":11271,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.430539+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:31.430539+0000","last_peered":"2024-09-10T20:29:31.430539+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:31.430539+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8829544,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11061,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11277","reported_seq":15456,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.219887+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:33.219887+0000","last_peered":"2024-09-10T20:29:33.219887+0000","last_clean":"2024-09-10T20:29:33.219887+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:33.219887+0000","last_undegraded":"2024-09-10T20:29:33.219887+0000","last_fullsized":"2024-09-10T20:29:33.219887+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9248768,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"42'11025","reported_seq":11350,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.703454+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:29.703454+0000","last_peered":"2024-09-10T20:29:29.703454+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:29.703454+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368736,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"42'10992","reported_seq":11219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.186613+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:29.186613+0000","last_peered":"2024-09-10T20:29:29.186613+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:29.186613+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9060352,"num_objects":1106,"num_object_clones":0,"num_object_copies":4424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11020","reported_seq":11355,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.217643+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:33.217643+0000","last_peered":"2024-09-10T20:29:33.217643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:33.217643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598688,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-783}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11194","reported_seq":11371,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.143713+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:33.143713+0000","last_peered":"2024-09-10T20:29:33.143713+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:33.143713+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9000096,"num_objects":1099,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-901}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'10903","reported_seq":14879,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:29:29.729612+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:29.729612+0000","last_peered":"2024-09-10T20:29:29.729612+0000","last_clean":"2024-09-10T20:29:29.729612+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:29.729612+0000","last_undegraded":"2024-09-10T20:29:29.729612+0000","last_fullsized":"2024-09-10T20:29:29.729612+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8993360,"num_objects":1098,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"43'11030","reported_seq":15008,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:29:31.440335+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:31.440335+0000","last_peered":"2024-09-10T20:29:31.440335+0000","last_clean":"2024-09-10T20:29:31.440335+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:31.440335+0000","last_undegraded":"2024-09-10T20:29:31.440335+0000","last_fullsized":"2024-09-10T20:29:31.440335+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109504,"num_objects":1112,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"42'10991","reported_seq":11269,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.680805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:29.680805+0000","last_peered":"2024-09-10T20:29:29.680805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:29.680805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9187056,"num_objects":1122,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"42'11022","reported_seq":11325,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:29.422021+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:29.422021+0000","last_peered":"2024-09-10T20:29:29.422021+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:29.422021+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362000,"num_objects":1143,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11021,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'10777","reported_seq":10967,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.262565+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:31.262565+0000","last_peered":"2024-09-10T20:29:31.262565+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:31.262565+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8632912,"num_objects":1054,"num_object_clones":0,"num_object_copies":4216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-849}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11041","reported_seq":15085,"reported_epoch":44,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:29:33.216099+0000","last_change":"2024-09-10T20:29:33.035436+0000","last_active":"2024-09-10T20:29:33.216099+0000","last_peered":"2024-09-10T20:29:33.216099+0000","last_clean":"2024-09-10T20:29:33.216099+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:33.216099+0000","last_undegraded":"2024-09-10T20:29:33.216099+0000","last_fullsized":"2024-09-10T20:29:33.216099+0000","mapping_epoch":43,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":99,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623264,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"43'11331","reported_seq":11543,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:31.377106+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:31.377106+0000","last_peered":"2024-09-10T20:29:31.377106+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:31.377106+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9155744,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11329,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11027","reported_seq":15132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.163591+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:33.163591+0000","last_peered":"2024-09-10T20:29:33.163591+0000","last_clean":"2024-09-10T20:29:33.163591+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:33.163591+0000","last_undegraded":"2024-09-10T20:29:33.163591+0000","last_fullsized":"2024-09-10T20:29:33.163591+0000","mapping_epoch":41,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":1096,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":143794024,"num_objects":17557,"num_object_clones":0,"num_object_copies":70228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147661,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286605312,"data_stored":286605312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712424,"kb_used_data":72648,"kb_used_omap":4,"kb_used_meta":639739,"kb_avail":93659416,"statfs":{"total":96636764160,"available":95907241984,"internally_reserved":0,"allocated":74391552,"data_stored":74169036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":655093456},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715840,"kb_used_data":71020,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93656000,"statfs":{"total":96636764160,"available":95903744000,"internally_reserved":0,"allocated":72724480,"data_stored":72501964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574700,"kb_used_data":69584,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93797140,"statfs":{"total":96636764160,"available":96048271360,"internally_reserved":0,"allocated":71254016,"data_stored":71021971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715136,"kb_used_data":69860,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93656704,"statfs":{"total":96636764160,"available":95904464896,"internally_reserved":0,"allocated":71536640,"data_stored":71304595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70627328,"data_stored":70627328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70668288,"data_stored":70668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71921664,"data_stored":71921664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73388032,"data_stored":73388032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:39.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:39.622 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:39.622 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:39.654 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-10T20:29:38.102710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139091816,"num_objects":16983,"num_object_clones":0,"num_object_copies":67932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148235,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2720268,"kb_used_data":275532,"kb_used_omap":14,"kb_used_meta":2444593,"kb_avail":374767092,"statfs":{"total":386547056640,"available":383761502208,"internally_reserved":0,"allocated":282144768,"data_stored":281216588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2503263798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,10,7,16,12,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9904128,"num_objects":-1209,"num_object_clones":0,"num_object_copies":-4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-70,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1209,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003391"},"pg_stats":[{"pgid":"1.f","version":"44'11084","reported_seq":11375,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:34.556028+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:34.556028+0000","last_peered":"2024-09-10T20:29:34.556028+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:34.556028+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8503296,"num_objects":1038,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11084,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11183","reported_seq":11324,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.092566+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:33.092566+0000","last_peered":"2024-09-10T20:29:33.092566+0000","last_clean":"2024-09-10T20:29:33.092566+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:33.092566+0000","last_undegraded":"2024-09-10T20:29:33.092566+0000","last_fullsized":"2024-09-10T20:29:33.092566+0000","mapping_epoch":35,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757248,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11119","reported_seq":11331,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.494448+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:36.494448+0000","last_peered":"2024-09-10T20:29:36.494448+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:36.494448+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8362600,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11277","reported_seq":15456,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.219887+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:33.219887+0000","last_peered":"2024-09-10T20:29:33.219887+0000","last_clean":"2024-09-10T20:29:33.219887+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:33.219887+0000","last_undegraded":"2024-09-10T20:29:33.219887+0000","last_fullsized":"2024-09-10T20:29:33.219887+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9248768,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11094","reported_seq":11423,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:34.478416+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:34.478416+0000","last_peered":"2024-09-10T20:29:34.478416+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:34.478416+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8803488,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11054","reported_seq":11285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:34.582153+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:34.582153+0000","last_peered":"2024-09-10T20:29:34.582153+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:34.582153+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8552448,"num_objects":1044,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11020","reported_seq":11355,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.217643+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:33.217643+0000","last_peered":"2024-09-10T20:29:33.217643+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:33.217643+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598688,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-783}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11194","reported_seq":11371,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:33.143713+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:33.143713+0000","last_peered":"2024-09-10T20:29:33.143713+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:33.143713+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9000096,"num_objects":1099,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-901}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'10971","reported_seq":14952,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:34.639824+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:34.639824+0000","last_peered":"2024-09-10T20:29:34.639824+0000","last_clean":"2024-09-10T20:29:34.639824+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:34.639824+0000","last_undegraded":"2024-09-10T20:29:34.639824+0000","last_fullsized":"2024-09-10T20:29:34.639824+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8436304,"num_objects":1030,"num_object_clones":0,"num_object_copies":4120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7524,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11081","reported_seq":15061,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:36.383639+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:36.383639+0000","last_peered":"2024-09-10T20:29:36.383639+0000","last_clean":"2024-09-10T20:29:36.383639+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:36.383639+0000","last_undegraded":"2024-09-10T20:29:36.383639+0000","last_fullsized":"2024-09-10T20:29:36.383639+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3719,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11040","reported_seq":11322,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:34.746833+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:34.746833+0000","last_peered":"2024-09-10T20:29:34.746833+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:34.746833+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8785648,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-783}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11073","reported_seq":11380,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:34.681246+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:34.681246+0000","last_peered":"2024-09-10T20:29:34.681246+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:34.681246+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8944208,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11072,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-768}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10829","reported_seq":11021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.419677+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:36.419677+0000","last_peered":"2024-09-10T20:29:36.419677+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:36.419677+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8206928,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11041","reported_seq":15085,"reported_epoch":44,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T20:29:33.216099+0000","last_change":"2024-09-10T20:29:33.035436+0000","last_active":"2024-09-10T20:29:33.216099+0000","last_peered":"2024-09-10T20:29:33.216099+0000","last_clean":"2024-09-10T20:29:33.216099+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:33.216099+0000","last_undegraded":"2024-09-10T20:29:33.216099+0000","last_fullsized":"2024-09-10T20:29:33.216099+0000","mapping_epoch":43,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":99,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623264,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11399","reported_seq":11612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.501011+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:36.501011+0000","last_peered":"2024-09-10T20:29:36.501011+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:36.501011+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598688,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11027","reported_seq":15132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:33.163591+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:33.163591+0000","last_peered":"2024-09-10T20:29:33.163591+0000","last_clean":"2024-09-10T20:29:33.163591+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:33.163591+0000","last_undegraded":"2024-09-10T20:29:33.163591+0000","last_fullsized":"2024-09-10T20:29:33.163591+0000","mapping_epoch":41,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":1096,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":139091816,"num_objects":16983,"num_object_clones":0,"num_object_copies":67932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148235,"num_write_kb":578863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278716416,"data_stored":278716416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714156,"kb_used_data":68800,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93657684,"statfs":{"total":96636764160,"available":95905468416,"internally_reserved":0,"allocated":70451200,"data_stored":70219155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716276,"kb_used_data":67288,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93655564,"statfs":{"total":96636764160,"available":95903297536,"internally_reserved":0,"allocated":68902912,"data_stored":68670867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574700,"kb_used_data":69584,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93797140,"statfs":{"total":96636764160,"available":96048271360,"internally_reserved":0,"allocated":71254016,"data_stored":71021971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715136,"kb_used_data":69860,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93656704,"statfs":{"total":96636764160,"available":95904464896,"internally_reserved":0,"allocated":71536640,"data_stored":71304595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70627328,"data_stored":70627328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70668288,"data_stored":70668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68022272,"data_stored":68022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69398528,"data_stored":69398528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:39.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:40.272 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:40.272 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:40.298 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-10T20:29:40.105700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":133040840,"num_objects":16244,"num_object_clones":0,"num_object_copies":64976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137926,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738924,"kb_used_data":263684,"kb_used_omap":14,"kb_used_meta":2475121,"kb_avail":374748436,"statfs":{"total":386547056640,"available":383742398464,"internally_reserved":0,"allocated":270012416,"data_stored":269084236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2534524470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,4,15,18,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15955104,"num_objects":-1948,"num_object_clones":0,"num_object_copies":-7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9100,"num_write_kb":-48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.878860"},"pg_stats":[{"pgid":"1.f","version":"44'11139","reported_seq":11430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.716593+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:39.716593+0000","last_peered":"2024-09-10T20:29:39.716593+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:39.716593+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8052736,"num_objects":983,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11249","reported_seq":11389,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:38.210298+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:38.210298+0000","last_peered":"2024-09-10T20:29:38.210298+0000","last_clean":"2024-09-10T20:29:38.210298+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:38.210298+0000","last_undegraded":"2024-09-10T20:29:38.210298+0000","last_fullsized":"2024-09-10T20:29:38.210298+0000","mapping_epoch":35,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8216576,"num_objects":1003,"num_object_clones":0,"num_object_copies":4012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11119","reported_seq":11331,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.494448+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:36.494448+0000","last_peered":"2024-09-10T20:29:36.494448+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:36.494448+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8362600,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11339","reported_seq":15518,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:38.207269+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:38.207269+0000","last_peered":"2024-09-10T20:29:38.207269+0000","last_clean":"2024-09-10T20:29:38.207269+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:38.207269+0000","last_undegraded":"2024-09-10T20:29:38.207269+0000","last_fullsized":"2024-09-10T20:29:38.207269+0000","mapping_epoch":37,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8740864,"num_objects":1067,"num_object_clones":0,"num_object_copies":4268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11154","reported_seq":11482,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.761378+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:39.761378+0000","last_peered":"2024-09-10T20:29:39.761378+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:39.761378+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8311968,"num_objects":1015,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11114","reported_seq":11345,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.681873+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:39.681873+0000","last_peered":"2024-09-10T20:29:39.681873+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:39.681873+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11073","reported_seq":11554,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:38.164105+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:38.164105+0000","last_peered":"2024-09-10T20:29:38.164105+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:38.164105+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8164512,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-764}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11263","reported_seq":11439,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:38.269308+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:38.269308+0000","last_peered":"2024-09-10T20:29:38.269308+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:38.269308+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8434848,"num_objects":1030,"num_object_clones":0,"num_object_copies":4120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11261,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11030","reported_seq":15011,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:39.500965+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:39.500965+0000","last_peered":"2024-09-10T20:29:39.500965+0000","last_clean":"2024-09-10T20:29:39.500965+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:39.500965+0000","last_undegraded":"2024-09-10T20:29:39.500965+0000","last_fullsized":"2024-09-10T20:29:39.500965+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7952976,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11081","reported_seq":15061,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:36.383639+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:36.383639+0000","last_peered":"2024-09-10T20:29:36.383639+0000","last_clean":"2024-09-10T20:29:36.383639+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:36.383639+0000","last_undegraded":"2024-09-10T20:29:36.383639+0000","last_fullsized":"2024-09-10T20:29:36.383639+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3719,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11104","reported_seq":11386,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.650755+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:39.650755+0000","last_peered":"2024-09-10T20:29:39.650755+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:39.650755+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8261360,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11101,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11131","reported_seq":11438,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.585232+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:39.585232+0000","last_peered":"2024-09-10T20:29:39.585232+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:39.585232+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8469072,"num_objects":1034,"num_object_clones":0,"num_object_copies":4136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11130,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10829","reported_seq":11021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.419677+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:36.419677+0000","last_peered":"2024-09-10T20:29:36.419677+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:36.419677+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8206928,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11103","reported_seq":15163,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:38.247254+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:38.247254+0000","last_peered":"2024-09-10T20:29:38.247254+0000","last_clean":"2024-09-10T20:29:38.247254+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:38.247254+0000","last_undegraded":"2024-09-10T20:29:38.247254+0000","last_fullsized":"2024-09-10T20:29:38.247254+0000","mapping_epoch":43,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101888,"num_objects":989,"num_object_clones":0,"num_object_copies":3956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11399","reported_seq":11612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.501011+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:36.501011+0000","last_peered":"2024-09-10T20:29:36.501011+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:36.501011+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598688,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11096","reported_seq":15200,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:37.902094+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:37.902094+0000","last_peered":"2024-09-10T20:29:37.902094+0000","last_clean":"2024-09-10T20:29:37.902094+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:37.902094+0000","last_undegraded":"2024-09-10T20:29:37.902094+0000","last_fullsized":"2024-09-10T20:29:37.902094+0000","mapping_epoch":41,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8413184,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":133040840,"num_objects":16244,"num_object_clones":0,"num_object_copies":64976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137926,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266469376,"data_stored":266469376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714344,"kb_used_data":64828,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93657496,"statfs":{"total":96636764160,"available":95905275904,"internally_reserved":0,"allocated":66383872,"data_stored":66151827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716276,"kb_used_data":67288,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93655564,"statfs":{"total":96636764160,"available":95903297536,"internally_reserved":0,"allocated":68902912,"data_stored":68670867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592936,"kb_used_data":65624,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93778904,"statfs":{"total":96636764160,"available":96029597696,"internally_reserved":0,"allocated":67198976,"data_stored":66966931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215232,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715368,"kb_used_data":65944,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93656472,"statfs":{"total":96636764160,"available":95904227328,"internally_reserved":0,"allocated":67526656,"data_stored":67294611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66551808,"data_stored":66551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66609152,"data_stored":66609152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68022272,"data_stored":68022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65286144,"data_stored":65286144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:40.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:40.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11139", 'reported_seq': 11430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.716593+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:39.716593+0000', 'last_peered': '2024-09-10T20:29:39.716593+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:39.716593+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8052736, 'num_objects': 983, 'num_object_clones': 0, 'num_object_copies': 3932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11139, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -872}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:40.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:40.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11119", 'reported_seq': 11331, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:36.494448+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:36.494448+0000', 'last_peered': '2024-09-10T20:29:36.494448+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:36.494448+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8362600, 'num_objects': 1022, 'num_object_clones': 0, 'num_object_copies': 4088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1022, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11118, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -886}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:40.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:40.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11154", 'reported_seq': 11482, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.761378+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:39.761378+0000', 'last_peered': '2024-09-10T20:29:39.761378+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:39.761378+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8311968, 'num_objects': 1015, 'num_object_clones': 0, 'num_object_copies': 4060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11152, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -840}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:40.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:40.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11114", 'reported_seq': 11345, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.681873+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:39.681873+0000', 'last_peered': '2024-09-10T20:29:39.681873+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:39.681873+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8060928, 'num_objects': 984, 'num_object_clones': 0, 'num_object_copies': 3936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11114, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:40.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11073", 'reported_seq': 11554, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:38.164105+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:29:38.164105+0000', 'last_peered': '2024-09-10T20:29:38.164105+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:38.164105+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8164512, 'num_objects': 997, 'num_object_clones': 0, 'num_object_copies': 3988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1761, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11071, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 196, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -764}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11263", 'reported_seq': 11439, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:38.269308+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:38.269308+0000', 'last_peered': '2024-09-10T20:29:38.269308+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:38.269308+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8434848, 'num_objects': 1030, 'num_object_clones': 0, 'num_object_copies': 4120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11261, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11104", 'reported_seq': 11386, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.650755+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:39.650755+0000', 'last_peered': '2024-09-10T20:29:39.650755+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:39.650755+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8261360, 'num_objects': 1009, 'num_object_clones': 0, 'num_object_copies': 4036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11101, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -847}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11131", 'reported_seq': 11438, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.585232+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:39.585232+0000', 'last_peered': '2024-09-10T20:29:39.585232+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:39.585232+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8469072, 'num_objects': 1034, 'num_object_clones': 0, 'num_object_copies': 4136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11130, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -826}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:40.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:40.302 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'10829", 'reported_seq': 11021, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:36.419677+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:36.419677+0000', 'last_peered': '2024-09-10T20:29:36.419677+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:36.419677+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8206928, 'num_objects': 1002, 'num_object_clones': 0, 'num_object_copies': 4008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10828, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:40.302 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:40.302 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11399", 'reported_seq': 11612, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:36.501011+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:36.501011+0000', 'last_peered': '2024-09-10T20:29:36.501011+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:36.501011+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8598688, 'num_objects': 1050, 'num_object_clones': 0, 'num_object_copies': 4200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11397, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:40.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:41.083 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:41.083 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18775/64976 objects degraded (28.895%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":669,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16244,"data_bytes":133040840,"bytes_used":2804658176,"bytes_avail":383742398464,"bytes_total":386547056640,"degraded_objects":18775,"degraded_total":64976,"degraded_ratio":0.28895284412706229},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:41.114 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:29:41.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:41.716 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:41.717 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:41.747 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-10T20:29:40.105700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":133040840,"num_objects":16244,"num_object_clones":0,"num_object_copies":64976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137926,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738924,"kb_used_data":263684,"kb_used_omap":14,"kb_used_meta":2475121,"kb_avail":374748436,"statfs":{"total":386547056640,"available":383742398464,"internally_reserved":0,"allocated":270012416,"data_stored":269084236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2534524470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,4,15,18,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15955104,"num_objects":-1948,"num_object_clones":0,"num_object_copies":-7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9100,"num_write_kb":-48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.878860"},"pg_stats":[{"pgid":"1.f","version":"44'11139","reported_seq":11430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.716593+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:39.716593+0000","last_peered":"2024-09-10T20:29:39.716593+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:39.716593+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8052736,"num_objects":983,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11249","reported_seq":11389,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:38.210298+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:38.210298+0000","last_peered":"2024-09-10T20:29:38.210298+0000","last_clean":"2024-09-10T20:29:38.210298+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:38.210298+0000","last_undegraded":"2024-09-10T20:29:38.210298+0000","last_fullsized":"2024-09-10T20:29:38.210298+0000","mapping_epoch":35,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8216576,"num_objects":1003,"num_object_clones":0,"num_object_copies":4012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11119","reported_seq":11331,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.494448+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:36.494448+0000","last_peered":"2024-09-10T20:29:36.494448+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:36.494448+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8362600,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11339","reported_seq":15518,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:38.207269+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:38.207269+0000","last_peered":"2024-09-10T20:29:38.207269+0000","last_clean":"2024-09-10T20:29:38.207269+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:38.207269+0000","last_undegraded":"2024-09-10T20:29:38.207269+0000","last_fullsized":"2024-09-10T20:29:38.207269+0000","mapping_epoch":37,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8740864,"num_objects":1067,"num_object_clones":0,"num_object_copies":4268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11154","reported_seq":11482,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.761378+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:39.761378+0000","last_peered":"2024-09-10T20:29:39.761378+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:39.761378+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8311968,"num_objects":1015,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11114","reported_seq":11345,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.681873+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:39.681873+0000","last_peered":"2024-09-10T20:29:39.681873+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:39.681873+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11073","reported_seq":11554,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:38.164105+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:38.164105+0000","last_peered":"2024-09-10T20:29:38.164105+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:38.164105+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8164512,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-764}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11263","reported_seq":11439,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:38.269308+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:38.269308+0000","last_peered":"2024-09-10T20:29:38.269308+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:38.269308+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8434848,"num_objects":1030,"num_object_clones":0,"num_object_copies":4120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11261,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11030","reported_seq":15011,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:39.500965+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:39.500965+0000","last_peered":"2024-09-10T20:29:39.500965+0000","last_clean":"2024-09-10T20:29:39.500965+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:39.500965+0000","last_undegraded":"2024-09-10T20:29:39.500965+0000","last_fullsized":"2024-09-10T20:29:39.500965+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7952976,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11081","reported_seq":15061,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:36.383639+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:36.383639+0000","last_peered":"2024-09-10T20:29:36.383639+0000","last_clean":"2024-09-10T20:29:36.383639+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:36.383639+0000","last_undegraded":"2024-09-10T20:29:36.383639+0000","last_fullsized":"2024-09-10T20:29:36.383639+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3719,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11104","reported_seq":11386,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.650755+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:39.650755+0000","last_peered":"2024-09-10T20:29:39.650755+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:39.650755+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8261360,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11101,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11131","reported_seq":11438,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.585232+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:39.585232+0000","last_peered":"2024-09-10T20:29:39.585232+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:39.585232+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8469072,"num_objects":1034,"num_object_clones":0,"num_object_copies":4136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11130,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10829","reported_seq":11021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.419677+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:36.419677+0000","last_peered":"2024-09-10T20:29:36.419677+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:36.419677+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8206928,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11103","reported_seq":15163,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:38.247254+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:38.247254+0000","last_peered":"2024-09-10T20:29:38.247254+0000","last_clean":"2024-09-10T20:29:38.247254+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:38.247254+0000","last_undegraded":"2024-09-10T20:29:38.247254+0000","last_fullsized":"2024-09-10T20:29:38.247254+0000","mapping_epoch":43,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101888,"num_objects":989,"num_object_clones":0,"num_object_copies":3956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11399","reported_seq":11612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:36.501011+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:36.501011+0000","last_peered":"2024-09-10T20:29:36.501011+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:36.501011+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598688,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11096","reported_seq":15200,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:37.902094+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:37.902094+0000","last_peered":"2024-09-10T20:29:37.902094+0000","last_clean":"2024-09-10T20:29:37.902094+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:37.902094+0000","last_undegraded":"2024-09-10T20:29:37.902094+0000","last_fullsized":"2024-09-10T20:29:37.902094+0000","mapping_epoch":41,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8413184,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":133040840,"num_objects":16244,"num_object_clones":0,"num_object_copies":64976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137926,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266469376,"data_stored":266469376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714344,"kb_used_data":64828,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93657496,"statfs":{"total":96636764160,"available":95905275904,"internally_reserved":0,"allocated":66383872,"data_stored":66151827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716276,"kb_used_data":67288,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93655564,"statfs":{"total":96636764160,"available":95903297536,"internally_reserved":0,"allocated":68902912,"data_stored":68670867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592936,"kb_used_data":65624,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93778904,"statfs":{"total":96636764160,"available":96029597696,"internally_reserved":0,"allocated":67198976,"data_stored":66966931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215232,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715368,"kb_used_data":65944,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93656472,"statfs":{"total":96636764160,"available":95904227328,"internally_reserved":0,"allocated":67526656,"data_stored":67294611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66551808,"data_stored":66551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66609152,"data_stored":66609152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68022272,"data_stored":68022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65286144,"data_stored":65286144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:44.748 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:45.354 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:45.375 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-10T20:29:44.107375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127486664,"num_objects":15566,"num_object_clones":0,"num_object_copies":62264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138604,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722384,"kb_used_data":251300,"kb_used_omap":14,"kb_used_meta":2470961,"kb_avail":374764976,"statfs":{"total":386547056640,"available":383759335424,"internally_reserved":0,"allocated":257331200,"data_stored":256403020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2530264630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,9,15,17,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19494048,"num_objects":-2380,"num_object_clones":0,"num_object_copies":-9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8668,"num_write_kb":-48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1796,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.865711"},"pg_stats":[{"pgid":"1.f","version":"44'11139","reported_seq":11430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.716593+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:39.716593+0000","last_peered":"2024-09-10T20:29:39.716593+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:39.716593+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8052736,"num_objects":983,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11334","reported_seq":11475,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.363266+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:43.363266+0000","last_peered":"2024-09-10T20:29:43.363266+0000","last_clean":"2024-09-10T20:29:43.363266+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:43.363266+0000","last_undegraded":"2024-09-10T20:29:43.363266+0000","last_fullsized":"2024-09-10T20:29:43.363266+0000","mapping_epoch":35,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":918,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7775,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11189","reported_seq":11401,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.365102+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:41.365102+0000","last_peered":"2024-09-10T20:29:41.365102+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:41.365102+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7789160,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11188,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11415","reported_seq":15594,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.211139+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:43.211139+0000","last_peered":"2024-09-10T20:29:43.211139+0000","last_clean":"2024-09-10T20:29:43.211139+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:43.211139+0000","last_undegraded":"2024-09-10T20:29:43.211139+0000","last_fullsized":"2024-09-10T20:29:43.211139+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8118272,"num_objects":991,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11154","reported_seq":11482,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.761378+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:39.761378+0000","last_peered":"2024-09-10T20:29:39.761378+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:39.761378+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8311968,"num_objects":1015,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11114","reported_seq":11345,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.681873+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:39.681873+0000","last_peered":"2024-09-10T20:29:39.681873+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:39.681873+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11142","reported_seq":11705,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:43.316575+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:43.316575+0000","last_peered":"2024-09-10T20:29:43.316575+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:43.316575+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7599264,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-792}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11322","reported_seq":11499,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:42.932940+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:42.932940+0000","last_peered":"2024-09-10T20:29:42.932940+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:42.932940+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951520,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11320,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11030","reported_seq":15011,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:39.500965+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:39.500965+0000","last_peered":"2024-09-10T20:29:39.500965+0000","last_clean":"2024-09-10T20:29:39.500965+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:39.500965+0000","last_undegraded":"2024-09-10T20:29:39.500965+0000","last_fullsized":"2024-09-10T20:29:39.500965+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7952976,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11155","reported_seq":15135,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:41.583761+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:41.583761+0000","last_peered":"2024-09-10T20:29:41.583761+0000","last_clean":"2024-09-10T20:29:41.583761+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:41.583761+0000","last_undegraded":"2024-09-10T20:29:41.583761+0000","last_fullsized":"2024-09-10T20:29:41.583761+0000","mapping_epoch":39,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8085504,"num_objects":987,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11104","reported_seq":11386,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.650755+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:39.650755+0000","last_peered":"2024-09-10T20:29:39.650755+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:39.650755+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8261360,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11101,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11131","reported_seq":11438,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.585232+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:39.585232+0000","last_peered":"2024-09-10T20:29:39.585232+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:39.585232+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8469072,"num_objects":1034,"num_object_clones":0,"num_object_copies":4136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11130,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10889","reported_seq":11081,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.580672+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:41.580672+0000","last_peered":"2024-09-10T20:29:41.580672+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:41.580672+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7715408,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-961}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11155","reported_seq":15214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.263689+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:43.263689+0000","last_peered":"2024-09-10T20:29:43.263689+0000","last_clean":"2024-09-10T20:29:43.263689+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:43.263689+0000","last_undegraded":"2024-09-10T20:29:43.263689+0000","last_fullsized":"2024-09-10T20:29:43.263689+0000","mapping_epoch":43,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7675904,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11456","reported_seq":11669,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.519254+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:41.519254+0000","last_peered":"2024-09-10T20:29:41.519254+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:41.519254+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8131744,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11172","reported_seq":15277,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.137890+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:43.137890+0000","last_peered":"2024-09-10T20:29:43.137890+0000","last_clean":"2024-09-10T20:29:43.137890+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:43.137890+0000","last_undegraded":"2024-09-10T20:29:43.137890+0000","last_fullsized":"2024-09-10T20:29:43.137890+0000","mapping_epoch":41,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7790592,"num_objects":951,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":127486664,"num_objects":15566,"num_object_clones":0,"num_object_copies":62264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138604,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":253784064,"data_stored":253784064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714344,"kb_used_data":64828,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93657496,"statfs":{"total":96636764160,"available":95905275904,"internally_reserved":0,"allocated":66383872,"data_stored":66151827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716396,"kb_used_data":63260,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93655444,"statfs":{"total":96636764160,"available":95903174656,"internally_reserved":0,"allocated":64778240,"data_stored":64546195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,0,0,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576300,"kb_used_data":61468,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93795540,"statfs":{"total":96636764160,"available":96046632960,"internally_reserved":0,"allocated":62943232,"data_stored":62711187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527167186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,0,6,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715344,"kb_used_data":61744,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93656496,"statfs":{"total":96636764160,"available":95904251904,"internally_reserved":0,"allocated":63225856,"data_stored":62993811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62312448,"data_stored":62312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62324736,"data_stored":62324736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63860736,"data_stored":63860736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65286144,"data_stored":65286144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:45.376 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:46.009 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:46.036 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-10T20:29:44.107375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127486664,"num_objects":15566,"num_object_clones":0,"num_object_copies":62264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138604,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722384,"kb_used_data":251300,"kb_used_omap":14,"kb_used_meta":2470961,"kb_avail":374764976,"statfs":{"total":386547056640,"available":383759335424,"internally_reserved":0,"allocated":257331200,"data_stored":256403020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2530264630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,9,15,17,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19494048,"num_objects":-2380,"num_object_clones":0,"num_object_copies":-9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8668,"num_write_kb":-48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1796,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.865711"},"pg_stats":[{"pgid":"1.f","version":"44'11139","reported_seq":11430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.716593+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:39.716593+0000","last_peered":"2024-09-10T20:29:39.716593+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:39.716593+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8052736,"num_objects":983,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11334","reported_seq":11475,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.363266+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:43.363266+0000","last_peered":"2024-09-10T20:29:43.363266+0000","last_clean":"2024-09-10T20:29:43.363266+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:43.363266+0000","last_undegraded":"2024-09-10T20:29:43.363266+0000","last_fullsized":"2024-09-10T20:29:43.363266+0000","mapping_epoch":35,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":918,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7775,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11189","reported_seq":11401,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.365102+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:41.365102+0000","last_peered":"2024-09-10T20:29:41.365102+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:41.365102+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7789160,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11188,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11415","reported_seq":15594,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.211139+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:43.211139+0000","last_peered":"2024-09-10T20:29:43.211139+0000","last_clean":"2024-09-10T20:29:43.211139+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:43.211139+0000","last_undegraded":"2024-09-10T20:29:43.211139+0000","last_fullsized":"2024-09-10T20:29:43.211139+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8118272,"num_objects":991,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11154","reported_seq":11482,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.761378+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:39.761378+0000","last_peered":"2024-09-10T20:29:39.761378+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:39.761378+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8311968,"num_objects":1015,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11114","reported_seq":11345,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.681873+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:39.681873+0000","last_peered":"2024-09-10T20:29:39.681873+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:39.681873+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11142","reported_seq":11705,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:43.316575+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:43.316575+0000","last_peered":"2024-09-10T20:29:43.316575+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:43.316575+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7599264,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-792}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11322","reported_seq":11499,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:42.932940+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:42.932940+0000","last_peered":"2024-09-10T20:29:42.932940+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:42.932940+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951520,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11320,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11030","reported_seq":15011,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:39.500965+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:39.500965+0000","last_peered":"2024-09-10T20:29:39.500965+0000","last_clean":"2024-09-10T20:29:39.500965+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:39.500965+0000","last_undegraded":"2024-09-10T20:29:39.500965+0000","last_fullsized":"2024-09-10T20:29:39.500965+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7952976,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11155","reported_seq":15135,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:41.583761+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:41.583761+0000","last_peered":"2024-09-10T20:29:41.583761+0000","last_clean":"2024-09-10T20:29:41.583761+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:41.583761+0000","last_undegraded":"2024-09-10T20:29:41.583761+0000","last_fullsized":"2024-09-10T20:29:41.583761+0000","mapping_epoch":39,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8085504,"num_objects":987,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11104","reported_seq":11386,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.650755+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:39.650755+0000","last_peered":"2024-09-10T20:29:39.650755+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:39.650755+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8261360,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11101,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11131","reported_seq":11438,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:39.585232+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:39.585232+0000","last_peered":"2024-09-10T20:29:39.585232+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:39.585232+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8469072,"num_objects":1034,"num_object_clones":0,"num_object_copies":4136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11130,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10889","reported_seq":11081,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.580672+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:41.580672+0000","last_peered":"2024-09-10T20:29:41.580672+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:41.580672+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7715408,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-961}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11155","reported_seq":15214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.263689+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:43.263689+0000","last_peered":"2024-09-10T20:29:43.263689+0000","last_clean":"2024-09-10T20:29:43.263689+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:43.263689+0000","last_undegraded":"2024-09-10T20:29:43.263689+0000","last_fullsized":"2024-09-10T20:29:43.263689+0000","mapping_epoch":43,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7675904,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11456","reported_seq":11669,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.519254+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:41.519254+0000","last_peered":"2024-09-10T20:29:41.519254+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:41.519254+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8131744,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11172","reported_seq":15277,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.137890+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:43.137890+0000","last_peered":"2024-09-10T20:29:43.137890+0000","last_clean":"2024-09-10T20:29:43.137890+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:43.137890+0000","last_undegraded":"2024-09-10T20:29:43.137890+0000","last_fullsized":"2024-09-10T20:29:43.137890+0000","mapping_epoch":41,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7790592,"num_objects":951,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":127486664,"num_objects":15566,"num_object_clones":0,"num_object_copies":62264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138604,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":253784064,"data_stored":253784064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714344,"kb_used_data":64828,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93657496,"statfs":{"total":96636764160,"available":95905275904,"internally_reserved":0,"allocated":66383872,"data_stored":66151827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716396,"kb_used_data":63260,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93655444,"statfs":{"total":96636764160,"available":95903174656,"internally_reserved":0,"allocated":64778240,"data_stored":64546195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,0,0,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576300,"kb_used_data":61468,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93795540,"statfs":{"total":96636764160,"available":96046632960,"internally_reserved":0,"allocated":62943232,"data_stored":62711187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527167186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,0,6,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715344,"kb_used_data":61744,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93656496,"statfs":{"total":96636764160,"available":95904251904,"internally_reserved":0,"allocated":63225856,"data_stored":62993811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62312448,"data_stored":62312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62324736,"data_stored":62324736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63860736,"data_stored":63860736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65286144,"data_stored":65286144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11139", 'reported_seq': 11430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.716593+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:39.716593+0000', 'last_peered': '2024-09-10T20:29:39.716593+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:39.716593+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8052736, 'num_objects': 983, 'num_object_clones': 0, 'num_object_copies': 3932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11139, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -872}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11189", 'reported_seq': 11401, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:41.365102+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:41.365102+0000', 'last_peered': '2024-09-10T20:29:41.365102+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:41.365102+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7789160, 'num_objects': 952, 'num_object_clones': 0, 'num_object_copies': 3808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 952, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11188, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -956}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11154", 'reported_seq': 11482, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.761378+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:39.761378+0000', 'last_peered': '2024-09-10T20:29:39.761378+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:39.761378+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8311968, 'num_objects': 1015, 'num_object_clones': 0, 'num_object_copies': 4060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11152, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -840}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11114", 'reported_seq': 11345, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.681873+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:39.681873+0000', 'last_peered': '2024-09-10T20:29:39.681873+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:39.681873+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8060928, 'num_objects': 984, 'num_object_clones': 0, 'num_object_copies': 3936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11114, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:46.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11142", 'reported_seq': 11705, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:43.316575+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:29:43.316575+0000', 'last_peered': '2024-09-10T20:29:43.316575+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:43.316575+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7599264, 'num_objects': 928, 'num_object_clones': 0, 'num_object_copies': 3712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11140, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 237, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -792}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11322", 'reported_seq': 11499, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:42.932940+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:42.932940+0000', 'last_peered': '2024-09-10T20:29:42.932940+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:42.932940+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7951520, 'num_objects': 971, 'num_object_clones': 0, 'num_object_copies': 3884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11320, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11104", 'reported_seq': 11386, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.650755+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:39.650755+0000', 'last_peered': '2024-09-10T20:29:39.650755+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:39.650755+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8261360, 'num_objects': 1009, 'num_object_clones': 0, 'num_object_copies': 4036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11101, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -847}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11131", 'reported_seq': 11438, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:39.585232+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:39.585232+0000', 'last_peered': '2024-09-10T20:29:39.585232+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:39.585232+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8469072, 'num_objects': 1034, 'num_object_clones': 0, 'num_object_copies': 4136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11130, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -826}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'10889", 'reported_seq': 11081, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:41.580672+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:41.580672+0000', 'last_peered': '2024-09-10T20:29:41.580672+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:41.580672+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7715408, 'num_objects': 942, 'num_object_clones': 0, 'num_object_copies': 3768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10888, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -961}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:46.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11456", 'reported_seq': 11669, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:41.519254+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:41.519254+0000', 'last_peered': '2024-09-10T20:29:41.519254+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:41.519254+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8131744, 'num_objects': 993, 'num_object_clones': 0, 'num_object_copies': 3972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11454, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:46.040 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:46.785 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18734/60744 objects degraded (30.841%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":675,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":15186,"data_bytes":124373704,"bytes_used":2783350784,"bytes_avail":383763705856,"bytes_total":386547056640,"degraded_objects":18734,"degraded_total":60744,"degraded_ratio":0.30840906097721588},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:46.810 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:29:46.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:47.417 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:47.417 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:47.437 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-10T20:29:46.108112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":124373704,"num_objects":15186,"num_object_clones":0,"num_object_copies":60744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138984,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718116,"kb_used_data":247056,"kb_used_omap":14,"kb_used_meta":2470961,"kb_avail":374769244,"statfs":{"total":386547056640,"available":383763705856,"internally_reserved":0,"allocated":252985344,"data_stored":252057164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2530264630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,7,9,10,15,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19420320,"num_objects":-2371,"num_object_clones":0,"num_object_copies":-9484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8677,"num_write_kb":-48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1866,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006937"},"pg_stats":[{"pgid":"1.f","version":"44'11200","reported_seq":11491,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:44.698381+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:44.698381+0000","last_peered":"2024-09-10T20:29:44.698381+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:44.698381+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7553024,"num_objects":922,"num_object_clones":0,"num_object_copies":3688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11200,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11334","reported_seq":11475,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.363266+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:43.363266+0000","last_peered":"2024-09-10T20:29:43.363266+0000","last_clean":"2024-09-10T20:29:43.363266+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:43.363266+0000","last_undegraded":"2024-09-10T20:29:43.363266+0000","last_fullsized":"2024-09-10T20:29:43.363266+0000","mapping_epoch":35,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":918,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7775,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11189","reported_seq":11401,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.365102+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:41.365102+0000","last_peered":"2024-09-10T20:29:41.365102+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:41.365102+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7789160,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11188,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11415","reported_seq":15594,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.211139+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:43.211139+0000","last_peered":"2024-09-10T20:29:43.211139+0000","last_clean":"2024-09-10T20:29:43.211139+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:43.211139+0000","last_undegraded":"2024-09-10T20:29:43.211139+0000","last_fullsized":"2024-09-10T20:29:43.211139+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8118272,"num_objects":991,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11223","reported_seq":11552,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:44.873583+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:44.873583+0000","last_peered":"2024-09-10T20:29:44.873583+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:44.873583+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7746720,"num_objects":946,"num_object_clones":0,"num_object_copies":3784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11180","reported_seq":11411,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:44.700028+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:44.700028+0000","last_peered":"2024-09-10T20:29:44.700028+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:44.700028+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":918,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11180,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11142","reported_seq":11705,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:43.316575+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:43.316575+0000","last_peered":"2024-09-10T20:29:43.316575+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:43.316575+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7599264,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-792}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11322","reported_seq":11499,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:42.932940+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:42.932940+0000","last_peered":"2024-09-10T20:29:42.932940+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:42.932940+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951520,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11320,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11091","reported_seq":15072,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:44.778688+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:44.778688+0000","last_peered":"2024-09-10T20:29:44.778688+0000","last_clean":"2024-09-10T20:29:44.778688+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:44.778688+0000","last_undegraded":"2024-09-10T20:29:44.778688+0000","last_fullsized":"2024-09-10T20:29:44.778688+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7453264,"num_objects":910,"num_object_clones":0,"num_object_copies":3640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7644,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11155","reported_seq":15135,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:41.583761+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:41.583761+0000","last_peered":"2024-09-10T20:29:41.583761+0000","last_clean":"2024-09-10T20:29:41.583761+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:41.583761+0000","last_undegraded":"2024-09-10T20:29:41.583761+0000","last_fullsized":"2024-09-10T20:29:41.583761+0000","mapping_epoch":39,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8085504,"num_objects":987,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11170","reported_seq":11451,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:44.834507+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:44.834507+0000","last_peered":"2024-09-10T20:29:44.834507+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:44.834507+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7720688,"num_objects":943,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-913}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11188","reported_seq":11494,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:44.849089+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:44.849089+0000","last_peered":"2024-09-10T20:29:44.849089+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:44.849089+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8002128,"num_objects":977,"num_object_clones":0,"num_object_copies":3908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-883}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10889","reported_seq":11081,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.580672+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:41.580672+0000","last_peered":"2024-09-10T20:29:41.580672+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:41.580672+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7715408,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-961}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11155","reported_seq":15214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.263689+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:43.263689+0000","last_peered":"2024-09-10T20:29:43.263689+0000","last_clean":"2024-09-10T20:29:43.263689+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:43.263689+0000","last_undegraded":"2024-09-10T20:29:43.263689+0000","last_fullsized":"2024-09-10T20:29:43.263689+0000","mapping_epoch":43,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7675904,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11456","reported_seq":11669,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:41.519254+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:41.519254+0000","last_peered":"2024-09-10T20:29:41.519254+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:41.519254+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8131744,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11172","reported_seq":15277,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:43.137890+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:43.137890+0000","last_peered":"2024-09-10T20:29:43.137890+0000","last_clean":"2024-09-10T20:29:43.137890+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:43.137890+0000","last_undegraded":"2024-09-10T20:29:43.137890+0000","last_fullsized":"2024-09-10T20:29:43.137890+0000","mapping_epoch":41,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7790592,"num_objects":951,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":124373704,"num_objects":15186,"num_object_clones":0,"num_object_copies":60744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138984,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249483264,"data_stored":249483264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710076,"kb_used_data":60584,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93661764,"statfs":{"total":96636764160,"available":95909646336,"internally_reserved":0,"allocated":62038016,"data_stored":61805971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716396,"kb_used_data":63260,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93655444,"statfs":{"total":96636764160,"available":95903174656,"internally_reserved":0,"allocated":64778240,"data_stored":64546195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,0,0,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576300,"kb_used_data":61468,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93795540,"statfs":{"total":96636764160,"available":96046632960,"internally_reserved":0,"allocated":62943232,"data_stored":62711187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527167186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,0,6,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715344,"kb_used_data":61744,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93656496,"statfs":{"total":96636764160,"available":95904251904,"internally_reserved":0,"allocated":63225856,"data_stored":62993811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62312448,"data_stored":62312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62324736,"data_stored":62324736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63860736,"data_stored":63860736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60985344,"data_stored":60985344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:50.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:51.075 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:51.096 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-10T20:29:50.109412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116222664,"num_objects":14191,"num_object_clones":0,"num_object_copies":56764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139979,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757504,"kb_used_data":230896,"kb_used_omap":14,"kb_used_meta":2526513,"kb_avail":374729856,"statfs":{"total":386547056640,"available":383723372544,"internally_reserved":0,"allocated":236437504,"data_stored":235509324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2587149878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,8,6,10,27,1,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22869152,"num_objects":-2792,"num_object_clones":0,"num_object_copies":-11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8256,"num_write_kb":-48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1810,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006701"},"pg_stats":[{"pgid":"1.f","version":"44'11264","reported_seq":11555,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.797483+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:49.797483+0000","last_peered":"2024-09-10T20:29:49.797483+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:49.797483+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7028736,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11264,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11401","reported_seq":11541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.174752+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:48.174752+0000","last_peered":"2024-09-10T20:29:48.174752+0000","last_clean":"2024-09-10T20:29:48.174752+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:48.174752+0000","last_undegraded":"2024-09-10T20:29:48.174752+0000","last_fullsized":"2024-09-10T20:29:48.174752+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6971392,"num_objects":851,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7842,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11253","reported_seq":11465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:46.403218+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:46.403218+0000","last_peered":"2024-09-10T20:29:46.403218+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:46.403218+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7264872,"num_objects":888,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11252,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1020}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11474","reported_seq":15653,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.326823+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:48.326823+0000","last_peered":"2024-09-10T20:29:48.326823+0000","last_clean":"2024-09-10T20:29:48.326823+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:48.326823+0000","last_undegraded":"2024-09-10T20:29:48.326823+0000","last_fullsized":"2024-09-10T20:29:48.326823+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7634944,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11278","reported_seq":11608,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.864804+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:49.864804+0000","last_peered":"2024-09-10T20:29:49.864804+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:49.864804+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7296160,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11238","reported_seq":11469,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.653542+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:49.653542+0000","last_peered":"2024-09-10T20:29:49.653542+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:49.653542+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7045120,"num_objects":860,"num_object_clones":0,"num_object_copies":3440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11201","reported_seq":11876,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:48.298458+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:48.298458+0000","last_peered":"2024-09-10T20:29:48.298458+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:48.298458+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7115936,"num_objects":869,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-795}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11387","reported_seq":11564,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:48.122694+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:48.122694+0000","last_peered":"2024-09-10T20:29:48.122694+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:48.122694+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419040,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11385,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1094}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11169","reported_seq":15150,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:49.642586+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:49.642586+0000","last_peered":"2024-09-10T20:29:49.642586+0000","last_clean":"2024-09-10T20:29:49.642586+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:49.642586+0000","last_undegraded":"2024-09-10T20:29:49.642586+0000","last_fullsized":"2024-09-10T20:29:49.642586+0000","mapping_epoch":33,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6814288,"num_objects":832,"num_object_clones":0,"num_object_copies":3328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11215","reported_seq":15195,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:46.279410+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:46.279410+0000","last_peered":"2024-09-10T20:29:46.279410+0000","last_clean":"2024-09-10T20:29:46.279410+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:46.279410+0000","last_undegraded":"2024-09-10T20:29:46.279410+0000","last_fullsized":"2024-09-10T20:29:46.279410+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7593984,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11228","reported_seq":11509,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.822620+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:49.822620+0000","last_peered":"2024-09-10T20:29:49.822620+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:49.822620+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7245552,"num_objects":885,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11253","reported_seq":11560,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.796417+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:49.796417+0000","last_peered":"2024-09-10T20:29:49.796417+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:49.796417+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7469648,"num_objects":912,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-948}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10951","reported_seq":11143,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:46.364792+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:46.364792+0000","last_peered":"2024-09-10T20:29:46.364792+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:46.364792+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7207504,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1023}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11216","reported_seq":15275,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.303661+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:48.303661+0000","last_peered":"2024-09-10T20:29:48.303661+0000","last_clean":"2024-09-10T20:29:48.303661+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:48.303661+0000","last_undegraded":"2024-09-10T20:29:48.303661+0000","last_fullsized":"2024-09-10T20:29:48.303661+0000","mapping_epoch":43,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176192,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11512","reported_seq":11726,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:46.076181+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:46.076181+0000","last_peered":"2024-09-10T20:29:46.076181+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:46.076181+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7672992,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11236","reported_seq":15341,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.218840+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:48.218840+0000","last_peered":"2024-09-10T20:29:48.218840+0000","last_clean":"2024-09-10T20:29:48.218840+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:48.218840+0000","last_undegraded":"2024-09-10T20:29:48.218840+0000","last_fullsized":"2024-09-10T20:29:48.218840+0000","mapping_epoch":41,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7266304,"num_objects":887,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":116222664,"num_objects":14191,"num_object_clones":0,"num_object_copies":56764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139979,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":232820736,"data_stored":232820736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728372,"kb_used_data":56676,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93643468,"statfs":{"total":96636764160,"available":95890911232,"internally_reserved":0,"allocated":58036224,"data_stored":57804179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717728,"kb_used_data":59012,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93654112,"statfs":{"total":96636764160,"available":95901810688,"internally_reserved":0,"allocated":60428288,"data_stored":60196243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594544,"kb_used_data":57500,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93777296,"statfs":{"total":96636764160,"available":96027951104,"internally_reserved":0,"allocated":58880000,"data_stored":58647955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,8,0,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716860,"kb_used_data":57708,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93654980,"statfs":{"total":96636764160,"available":95902699520,"internally_reserved":0,"allocated":59092992,"data_stored":58860947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58228736,"data_stored":58228736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58130432,"data_stored":58130432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":59600896,"data_stored":59600896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56860672,"data_stored":56860672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:51.098 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:51.677 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:51.679 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:51.705 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-10T20:29:50.109412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116222664,"num_objects":14191,"num_object_clones":0,"num_object_copies":56764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139979,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757504,"kb_used_data":230896,"kb_used_omap":14,"kb_used_meta":2526513,"kb_avail":374729856,"statfs":{"total":386547056640,"available":383723372544,"internally_reserved":0,"allocated":236437504,"data_stored":235509324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2587149878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,8,6,10,27,1,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22869152,"num_objects":-2792,"num_object_clones":0,"num_object_copies":-11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8256,"num_write_kb":-48372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1810,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006701"},"pg_stats":[{"pgid":"1.f","version":"44'11264","reported_seq":11555,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.797483+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:49.797483+0000","last_peered":"2024-09-10T20:29:49.797483+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:49.797483+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7028736,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11264,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11401","reported_seq":11541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.174752+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:48.174752+0000","last_peered":"2024-09-10T20:29:48.174752+0000","last_clean":"2024-09-10T20:29:48.174752+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:48.174752+0000","last_undegraded":"2024-09-10T20:29:48.174752+0000","last_fullsized":"2024-09-10T20:29:48.174752+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6971392,"num_objects":851,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7842,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11253","reported_seq":11465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:46.403218+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:46.403218+0000","last_peered":"2024-09-10T20:29:46.403218+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:46.403218+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7264872,"num_objects":888,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11252,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1020}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11474","reported_seq":15653,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.326823+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:48.326823+0000","last_peered":"2024-09-10T20:29:48.326823+0000","last_clean":"2024-09-10T20:29:48.326823+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:48.326823+0000","last_undegraded":"2024-09-10T20:29:48.326823+0000","last_fullsized":"2024-09-10T20:29:48.326823+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7634944,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11278","reported_seq":11608,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.864804+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:49.864804+0000","last_peered":"2024-09-10T20:29:49.864804+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:49.864804+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7296160,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11238","reported_seq":11469,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.653542+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:49.653542+0000","last_peered":"2024-09-10T20:29:49.653542+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:49.653542+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7045120,"num_objects":860,"num_object_clones":0,"num_object_copies":3440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11201","reported_seq":11876,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:48.298458+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:48.298458+0000","last_peered":"2024-09-10T20:29:48.298458+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:48.298458+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7115936,"num_objects":869,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-795}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11387","reported_seq":11564,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:48.122694+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:48.122694+0000","last_peered":"2024-09-10T20:29:48.122694+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:48.122694+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419040,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11385,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1094}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11169","reported_seq":15150,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:49.642586+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:49.642586+0000","last_peered":"2024-09-10T20:29:49.642586+0000","last_clean":"2024-09-10T20:29:49.642586+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:49.642586+0000","last_undegraded":"2024-09-10T20:29:49.642586+0000","last_fullsized":"2024-09-10T20:29:49.642586+0000","mapping_epoch":33,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6814288,"num_objects":832,"num_object_clones":0,"num_object_copies":3328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11215","reported_seq":15195,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:46.279410+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:46.279410+0000","last_peered":"2024-09-10T20:29:46.279410+0000","last_clean":"2024-09-10T20:29:46.279410+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:46.279410+0000","last_undegraded":"2024-09-10T20:29:46.279410+0000","last_fullsized":"2024-09-10T20:29:46.279410+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7593984,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11228","reported_seq":11509,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.822620+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:49.822620+0000","last_peered":"2024-09-10T20:29:49.822620+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:49.822620+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7245552,"num_objects":885,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11253","reported_seq":11560,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.796417+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:49.796417+0000","last_peered":"2024-09-10T20:29:49.796417+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:49.796417+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7469648,"num_objects":912,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-948}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'10951","reported_seq":11143,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:46.364792+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:46.364792+0000","last_peered":"2024-09-10T20:29:46.364792+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:46.364792+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7207504,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1023}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11216","reported_seq":15275,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.303661+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:48.303661+0000","last_peered":"2024-09-10T20:29:48.303661+0000","last_clean":"2024-09-10T20:29:48.303661+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:48.303661+0000","last_undegraded":"2024-09-10T20:29:48.303661+0000","last_fullsized":"2024-09-10T20:29:48.303661+0000","mapping_epoch":43,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176192,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11512","reported_seq":11726,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:46.076181+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:46.076181+0000","last_peered":"2024-09-10T20:29:46.076181+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:46.076181+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7672992,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11236","reported_seq":15341,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.218840+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:48.218840+0000","last_peered":"2024-09-10T20:29:48.218840+0000","last_clean":"2024-09-10T20:29:48.218840+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:48.218840+0000","last_undegraded":"2024-09-10T20:29:48.218840+0000","last_fullsized":"2024-09-10T20:29:48.218840+0000","mapping_epoch":41,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7266304,"num_objects":887,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":116222664,"num_objects":14191,"num_object_clones":0,"num_object_copies":56764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139979,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":232820736,"data_stored":232820736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728372,"kb_used_data":56676,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93643468,"statfs":{"total":96636764160,"available":95890911232,"internally_reserved":0,"allocated":58036224,"data_stored":57804179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717728,"kb_used_data":59012,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93654112,"statfs":{"total":96636764160,"available":95901810688,"internally_reserved":0,"allocated":60428288,"data_stored":60196243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594544,"kb_used_data":57500,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93777296,"statfs":{"total":96636764160,"available":96027951104,"internally_reserved":0,"allocated":58880000,"data_stored":58647955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,8,0,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716860,"kb_used_data":57708,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93654980,"statfs":{"total":96636764160,"available":95902699520,"internally_reserved":0,"allocated":59092992,"data_stored":58860947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58228736,"data_stored":58228736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58130432,"data_stored":58130432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":59600896,"data_stored":59600896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56860672,"data_stored":56860672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11264", 'reported_seq': 11555, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:49.797483+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:49.797483+0000', 'last_peered': '2024-09-10T20:29:49.797483+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:49.797483+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7028736, 'num_objects': 858, 'num_object_clones': 0, 'num_object_copies': 3432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11264, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -997}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11253", 'reported_seq': 11465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:46.403218+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:46.403218+0000', 'last_peered': '2024-09-10T20:29:46.403218+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:46.403218+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7264872, 'num_objects': 888, 'num_object_clones': 0, 'num_object_copies': 3552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 888, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11252, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1020}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11278", 'reported_seq': 11608, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:49.864804+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:49.864804+0000', 'last_peered': '2024-09-10T20:29:49.864804+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:49.864804+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7296160, 'num_objects': 891, 'num_object_clones': 0, 'num_object_copies': 3564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11276, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -964}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:51.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11238", 'reported_seq': 11469, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:49.653542+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:49.653542+0000', 'last_peered': '2024-09-10T20:29:49.653542+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:49.653542+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7045120, 'num_objects': 860, 'num_object_clones': 0, 'num_object_copies': 3440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11238, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -997}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11201", 'reported_seq': 11876, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:48.298458+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:29:48.298458+0000', 'last_peered': '2024-09-10T20:29:48.298458+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:48.298458+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7115936, 'num_objects': 869, 'num_object_clones': 0, 'num_object_copies': 3476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1664, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11199, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 293, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -795}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11387", 'reported_seq': 11564, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:48.122694+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:48.122694+0000', 'last_peered': '2024-09-10T20:29:48.122694+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:48.122694+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7419040, 'num_objects': 906, 'num_object_clones': 0, 'num_object_copies': 3624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11385, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1094}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11228", 'reported_seq': 11509, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:49.822620+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:49.822620+0000', 'last_peered': '2024-09-10T20:29:49.822620+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:49.822620+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7245552, 'num_objects': 885, 'num_object_clones': 0, 'num_object_copies': 3540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11225, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -971}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11253", 'reported_seq': 11560, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:49.796417+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:49.796417+0000', 'last_peered': '2024-09-10T20:29:49.796417+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:49.796417+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7469648, 'num_objects': 912, 'num_object_clones': 0, 'num_object_copies': 3648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11252, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'10951", 'reported_seq': 11143, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:46.364792+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:46.364792+0000', 'last_peered': '2024-09-10T20:29:46.364792+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:46.364792+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7207504, 'num_objects': 880, 'num_object_clones': 0, 'num_object_copies': 3520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10950, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1023}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:51.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11512", 'reported_seq': 11726, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:46.076181+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:46.076181+0000', 'last_peered': '2024-09-10T20:29:46.076181+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:46.076181+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7672992, 'num_objects': 937, 'num_object_clones': 0, 'num_object_copies': 3748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11510, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -971}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:51.708 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:52.430 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18678/55624 objects degraded (33.579%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":681,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13906,"data_bytes":113887944,"bytes_used":2823839744,"bytes_avail":383723216896,"bytes_total":386547056640,"degraded_objects":18678,"degraded_total":55624,"degraded_ratio":0.33579030634258594,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:52.451 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:52.451 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:53.079 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:53.080 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:53.110 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-10T20:29:52.110239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":113887944,"num_objects":13906,"num_object_clones":0,"num_object_copies":55624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140264,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160761,"ondisk_log_size":160761,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757656,"kb_used_data":226900,"kb_used_omap":14,"kb_used_meta":2530673,"kb_avail":374729704,"statfs":{"total":386547056640,"available":383723216896,"internally_reserved":0,"allocated":232345600,"data_stored":231417420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2591409718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,6,6,19,2,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19152896,"num_objects":-2338,"num_object_clones":0,"num_object_copies":-9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004538"},"pg_stats":[{"pgid":"1.f","version":"44'11264","reported_seq":11555,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.797483+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:49.797483+0000","last_peered":"2024-09-10T20:29:49.797483+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:49.797483+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7028736,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11264,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11401","reported_seq":11541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.174752+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:48.174752+0000","last_peered":"2024-09-10T20:29:48.174752+0000","last_clean":"2024-09-10T20:29:48.174752+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:48.174752+0000","last_undegraded":"2024-09-10T20:29:48.174752+0000","last_fullsized":"2024-09-10T20:29:48.174752+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6971392,"num_objects":851,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7842,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11322","reported_seq":11534,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.652184+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:51.652184+0000","last_peered":"2024-09-10T20:29:51.652184+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:51.652184+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6699624,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11321,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11474","reported_seq":15653,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.326823+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:48.326823+0000","last_peered":"2024-09-10T20:29:48.326823+0000","last_clean":"2024-09-10T20:29:48.326823+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:48.326823+0000","last_undegraded":"2024-09-10T20:29:48.326823+0000","last_fullsized":"2024-09-10T20:29:48.326823+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7634944,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11278","reported_seq":11608,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.864804+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:49.864804+0000","last_peered":"2024-09-10T20:29:49.864804+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:49.864804+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7296160,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11238","reported_seq":11469,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.653542+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:49.653542+0000","last_peered":"2024-09-10T20:29:49.653542+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:49.653542+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7045120,"num_objects":860,"num_object_clones":0,"num_object_copies":3440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11201","reported_seq":11876,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:48.298458+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:48.298458+0000","last_peered":"2024-09-10T20:29:48.298458+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:48.298458+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7115936,"num_objects":869,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-795}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11387","reported_seq":11564,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:48.122694+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:48.122694+0000","last_peered":"2024-09-10T20:29:48.122694+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:48.122694+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419040,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11385,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1094}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11169","reported_seq":15150,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:49.642586+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:49.642586+0000","last_peered":"2024-09-10T20:29:49.642586+0000","last_clean":"2024-09-10T20:29:49.642586+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:49.642586+0000","last_undegraded":"2024-09-10T20:29:49.642586+0000","last_fullsized":"2024-09-10T20:29:49.642586+0000","mapping_epoch":33,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6814288,"num_objects":832,"num_object_clones":0,"num_object_copies":3328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11280","reported_seq":15260,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:51.602567+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:51.602567+0000","last_peered":"2024-09-10T20:29:51.602567+0000","last_clean":"2024-09-10T20:29:51.602567+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:51.602567+0000","last_undegraded":"2024-09-10T20:29:51.602567+0000","last_fullsized":"2024-09-10T20:29:51.602567+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7061504,"num_objects":862,"num_object_clones":0,"num_object_copies":3448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11228","reported_seq":11509,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.822620+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:49.822620+0000","last_peered":"2024-09-10T20:29:49.822620+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:49.822620+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7245552,"num_objects":885,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11253","reported_seq":11560,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:49.796417+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:49.796417+0000","last_peered":"2024-09-10T20:29:49.796417+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:49.796417+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7469648,"num_objects":912,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-948}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11030","reported_seq":11222,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.679639+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:51.679639+0000","last_peered":"2024-09-10T20:29:51.679639+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:51.679639+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6560336,"num_objects":801,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11029,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11216","reported_seq":15275,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.303661+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:48.303661+0000","last_peered":"2024-09-10T20:29:48.303661+0000","last_clean":"2024-09-10T20:29:48.303661+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:48.303661+0000","last_undegraded":"2024-09-10T20:29:48.303661+0000","last_fullsized":"2024-09-10T20:29:48.303661+0000","mapping_epoch":43,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176192,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11584","reported_seq":11798,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.455631+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:51.455631+0000","last_peered":"2024-09-10T20:29:51.455631+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:51.455631+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7083168,"num_objects":865,"num_object_clones":0,"num_object_copies":3460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11236","reported_seq":15341,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:48.218840+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:48.218840+0000","last_peered":"2024-09-10T20:29:48.218840+0000","last_clean":"2024-09-10T20:29:48.218840+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:48.218840+0000","last_undegraded":"2024-09-10T20:29:48.218840+0000","last_fullsized":"2024-09-10T20:29:48.218840+0000","mapping_epoch":41,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7266304,"num_objects":887,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":113887944,"num_objects":13906,"num_object_clones":0,"num_object_copies":55624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140264,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":228515840,"data_stored":228515840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160761,"ondisk_log_size":160761,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728372,"kb_used_data":56676,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93643468,"statfs":{"total":96636764160,"available":95890911232,"internally_reserved":0,"allocated":58036224,"data_stored":57804179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717880,"kb_used_data":55016,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93653960,"statfs":{"total":96636764160,"available":95901655040,"internally_reserved":0,"allocated":56336384,"data_stored":56104339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594544,"kb_used_data":57500,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93777296,"statfs":{"total":96636764160,"available":96027951104,"internally_reserved":0,"allocated":58880000,"data_stored":58647955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,8,0,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716860,"kb_used_data":57708,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93654980,"statfs":{"total":96636764160,"available":95902699520,"internally_reserved":0,"allocated":59092992,"data_stored":58860947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58228736,"data_stored":58228736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58130432,"data_stored":58130432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55296000,"data_stored":55296000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56860672,"data_stored":56860672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:56.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:56.804 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:56.804 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:56.828 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-10T20:29:56.111504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107612872,"num_objects":13140,"num_object_clones":0,"num_object_copies":52560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141030,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741284,"kb_used_data":214680,"kb_used_omap":14,"kb_used_meta":2526513,"kb_avail":374746076,"statfs":{"total":386547056640,"available":383739981824,"internally_reserved":0,"allocated":219832320,"data_stored":218904140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2587149878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,10,17,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19873792,"num_objects":-2426,"num_object_clones":0,"num_object_copies":-9704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004128"},"pg_stats":[{"pgid":"1.f","version":"44'11322","reported_seq":11613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.573497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:54.573497+0000","last_peered":"2024-09-10T20:29:54.573497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:54.573497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":800,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11322,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11456","reported_seq":11597,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.307433+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:53.307433+0000","last_peered":"2024-09-10T20:29:53.307433+0000","last_clean":"2024-09-10T20:29:53.307433+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:53.307433+0000","last_undegraded":"2024-09-10T20:29:53.307433+0000","last_fullsized":"2024-09-10T20:29:53.307433+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6520832,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7897,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11322","reported_seq":11534,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.652184+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:51.652184+0000","last_peered":"2024-09-10T20:29:51.652184+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:51.652184+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6699624,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11321,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11547","reported_seq":15726,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.516890+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:53.516890+0000","last_peered":"2024-09-10T20:29:53.516890+0000","last_clean":"2024-09-10T20:29:53.516890+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:53.516890+0000","last_undegraded":"2024-09-10T20:29:53.516890+0000","last_fullsized":"2024-09-10T20:29:53.516890+0000","mapping_epoch":37,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7036928,"num_objects":859,"num_object_clones":0,"num_object_copies":3436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11336","reported_seq":11666,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.773873+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:54.773873+0000","last_peered":"2024-09-10T20:29:54.773873+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:54.773873+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6821024,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11334,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11294","reported_seq":11525,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.675381+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:54.675381+0000","last_peered":"2024-09-10T20:29:54.675381+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:54.675381+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1053}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11278","reported_seq":12058,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:53.404076+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:53.404076+0000","last_peered":"2024-09-10T20:29:53.404076+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:53.404076+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6485152,"num_objects":792,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11459","reported_seq":11636,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:53.438816+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:53.438816+0000","last_peered":"2024-09-10T20:29:53.438816+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:53.438816+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6829216,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11245","reported_seq":15226,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:54.654894+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:54.654894+0000","last_peered":"2024-09-10T20:29:54.654894+0000","last_clean":"2024-09-10T20:29:54.654894+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:54.654894+0000","last_undegraded":"2024-09-10T20:29:54.654894+0000","last_fullsized":"2024-09-10T20:29:54.654894+0000","mapping_epoch":33,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6191696,"num_objects":756,"num_object_clones":0,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11280","reported_seq":15260,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:51.602567+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:51.602567+0000","last_peered":"2024-09-10T20:29:51.602567+0000","last_clean":"2024-09-10T20:29:51.602567+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:51.602567+0000","last_undegraded":"2024-09-10T20:29:51.602567+0000","last_fullsized":"2024-09-10T20:29:51.602567+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7061504,"num_objects":862,"num_object_clones":0,"num_object_copies":3448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11292","reported_seq":11574,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.886678+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:54.886678+0000","last_peered":"2024-09-10T20:29:54.886678+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:54.886678+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6721264,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11319","reported_seq":11623,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.802711+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:54.802711+0000","last_peered":"2024-09-10T20:29:54.802711+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:54.802711+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6928976,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11318,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1014}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11030","reported_seq":11222,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.679639+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:51.679639+0000","last_peered":"2024-09-10T20:29:51.679639+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:51.679639+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6560336,"num_objects":801,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11029,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11275","reported_seq":15335,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.315087+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:53.315087+0000","last_peered":"2024-09-10T20:29:53.315087+0000","last_clean":"2024-09-10T20:29:53.315087+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:53.315087+0000","last_undegraded":"2024-09-10T20:29:53.315087+0000","last_fullsized":"2024-09-10T20:29:53.315087+0000","mapping_epoch":43,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11584","reported_seq":11798,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.455631+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:51.455631+0000","last_peered":"2024-09-10T20:29:51.455631+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:51.455631+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7083168,"num_objects":865,"num_object_clones":0,"num_object_copies":3460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11288","reported_seq":15392,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.413645+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:53.413645+0000","last_peered":"2024-09-10T20:29:53.413645+0000","last_clean":"2024-09-10T20:29:53.413645+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:53.413645+0000","last_undegraded":"2024-09-10T20:29:53.413645+0000","last_fullsized":"2024-09-10T20:29:53.413645+0000","mapping_epoch":41,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840320,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107612872,"num_objects":13140,"num_object_clones":0,"num_object_copies":52560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141030,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215900160,"data_stored":215900160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711856,"kb_used_data":52640,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93659984,"statfs":{"total":96636764160,"available":95907823616,"internally_reserved":0,"allocated":53903360,"data_stored":53671315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717880,"kb_used_data":55016,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93653960,"statfs":{"total":96636764160,"available":95901655040,"internally_reserved":0,"allocated":56336384,"data_stored":56104339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594592,"kb_used_data":53392,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93777248,"statfs":{"total":96636764160,"available":96027901952,"internally_reserved":0,"allocated":54673408,"data_stored":54441363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554168018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,4,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716956,"kb_used_data":53632,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93654884,"statfs":{"total":96636764160,"available":95902601216,"internally_reserved":0,"allocated":54919168,"data_stored":54687123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679210703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53972992,"data_stored":53972992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53891072,"data_stored":53891072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55296000,"data_stored":55296000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52740096,"data_stored":52740096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:56.830 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:57.409 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:57.409 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:57.441 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-10T20:29:56.111504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107612872,"num_objects":13140,"num_object_clones":0,"num_object_copies":52560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141030,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741284,"kb_used_data":214680,"kb_used_omap":14,"kb_used_meta":2526513,"kb_avail":374746076,"statfs":{"total":386547056640,"available":383739981824,"internally_reserved":0,"allocated":219832320,"data_stored":218904140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2587149878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,10,17,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19873792,"num_objects":-2426,"num_object_clones":0,"num_object_copies":-9704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004128"},"pg_stats":[{"pgid":"1.f","version":"44'11322","reported_seq":11613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.573497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:54.573497+0000","last_peered":"2024-09-10T20:29:54.573497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:54.573497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":800,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11322,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11456","reported_seq":11597,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.307433+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:53.307433+0000","last_peered":"2024-09-10T20:29:53.307433+0000","last_clean":"2024-09-10T20:29:53.307433+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:53.307433+0000","last_undegraded":"2024-09-10T20:29:53.307433+0000","last_fullsized":"2024-09-10T20:29:53.307433+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6520832,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7897,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11322","reported_seq":11534,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.652184+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:51.652184+0000","last_peered":"2024-09-10T20:29:51.652184+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:51.652184+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6699624,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11321,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11547","reported_seq":15726,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.516890+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:53.516890+0000","last_peered":"2024-09-10T20:29:53.516890+0000","last_clean":"2024-09-10T20:29:53.516890+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:53.516890+0000","last_undegraded":"2024-09-10T20:29:53.516890+0000","last_fullsized":"2024-09-10T20:29:53.516890+0000","mapping_epoch":37,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7036928,"num_objects":859,"num_object_clones":0,"num_object_copies":3436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11336","reported_seq":11666,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.773873+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:54.773873+0000","last_peered":"2024-09-10T20:29:54.773873+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:54.773873+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6821024,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11334,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11294","reported_seq":11525,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.675381+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:54.675381+0000","last_peered":"2024-09-10T20:29:54.675381+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:54.675381+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1053}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11278","reported_seq":12058,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:53.404076+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:53.404076+0000","last_peered":"2024-09-10T20:29:53.404076+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:53.404076+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6485152,"num_objects":792,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11459","reported_seq":11636,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:53.438816+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:53.438816+0000","last_peered":"2024-09-10T20:29:53.438816+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:53.438816+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6829216,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11245","reported_seq":15226,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:54.654894+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:54.654894+0000","last_peered":"2024-09-10T20:29:54.654894+0000","last_clean":"2024-09-10T20:29:54.654894+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:54.654894+0000","last_undegraded":"2024-09-10T20:29:54.654894+0000","last_fullsized":"2024-09-10T20:29:54.654894+0000","mapping_epoch":33,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6191696,"num_objects":756,"num_object_clones":0,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11280","reported_seq":15260,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:51.602567+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:51.602567+0000","last_peered":"2024-09-10T20:29:51.602567+0000","last_clean":"2024-09-10T20:29:51.602567+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:51.602567+0000","last_undegraded":"2024-09-10T20:29:51.602567+0000","last_fullsized":"2024-09-10T20:29:51.602567+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7061504,"num_objects":862,"num_object_clones":0,"num_object_copies":3448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11292","reported_seq":11574,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.886678+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:54.886678+0000","last_peered":"2024-09-10T20:29:54.886678+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:54.886678+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6721264,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11319","reported_seq":11623,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.802711+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:54.802711+0000","last_peered":"2024-09-10T20:29:54.802711+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:54.802711+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6928976,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11318,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1014}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11030","reported_seq":11222,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.679639+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:51.679639+0000","last_peered":"2024-09-10T20:29:51.679639+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:51.679639+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6560336,"num_objects":801,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11029,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11275","reported_seq":15335,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.315087+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:53.315087+0000","last_peered":"2024-09-10T20:29:53.315087+0000","last_clean":"2024-09-10T20:29:53.315087+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:53.315087+0000","last_undegraded":"2024-09-10T20:29:53.315087+0000","last_fullsized":"2024-09-10T20:29:53.315087+0000","mapping_epoch":43,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11584","reported_seq":11798,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:51.455631+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:51.455631+0000","last_peered":"2024-09-10T20:29:51.455631+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:51.455631+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7083168,"num_objects":865,"num_object_clones":0,"num_object_copies":3460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11288","reported_seq":15392,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.413645+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:53.413645+0000","last_peered":"2024-09-10T20:29:53.413645+0000","last_clean":"2024-09-10T20:29:53.413645+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:53.413645+0000","last_undegraded":"2024-09-10T20:29:53.413645+0000","last_fullsized":"2024-09-10T20:29:53.413645+0000","mapping_epoch":41,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840320,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107612872,"num_objects":13140,"num_object_clones":0,"num_object_copies":52560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141030,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215900160,"data_stored":215900160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711856,"kb_used_data":52640,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93659984,"statfs":{"total":96636764160,"available":95907823616,"internally_reserved":0,"allocated":53903360,"data_stored":53671315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717880,"kb_used_data":55016,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93653960,"statfs":{"total":96636764160,"available":95901655040,"internally_reserved":0,"allocated":56336384,"data_stored":56104339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594592,"kb_used_data":53392,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93777248,"statfs":{"total":96636764160,"available":96027901952,"internally_reserved":0,"allocated":54673408,"data_stored":54441363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554168018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,4,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716956,"kb_used_data":53632,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93654884,"statfs":{"total":96636764160,"available":95902601216,"internally_reserved":0,"allocated":54919168,"data_stored":54687123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679210703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53972992,"data_stored":53972992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53891072,"data_stored":53891072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55296000,"data_stored":55296000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52740096,"data_stored":52740096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:29:57.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11322", 'reported_seq': 11613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:54.573497+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:54.573497+0000', 'last_peered': '2024-09-10T20:29:54.573497+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:54.573497+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6553600, 'num_objects': 800, 'num_object_clones': 0, 'num_object_copies': 3200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11322, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1055}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11322", 'reported_seq': 11534, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:51.652184+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:29:51.652184+0000', 'last_peered': '2024-09-10T20:29:51.652184+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:29:51.652184+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6699624, 'num_objects': 819, 'num_object_clones': 0, 'num_object_copies': 3276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 819, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11321, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1089}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11336", 'reported_seq': 11666, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:54.773873+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:54.773873+0000', 'last_peered': '2024-09-10T20:29:54.773873+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:54.773873+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6821024, 'num_objects': 833, 'num_object_clones': 0, 'num_object_copies': 3332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11334, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1022}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11294", 'reported_seq': 11525, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:54.675381+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:54.675381+0000', 'last_peered': '2024-09-10T20:29:54.675381+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:54.675381+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6586368, 'num_objects': 804, 'num_object_clones': 0, 'num_object_copies': 3216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11294, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1053}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:29:57.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11278", 'reported_seq': 12058, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:53.404076+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:29:53.404076+0000', 'last_peered': '2024-09-10T20:29:53.404076+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:53.404076+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6485152, 'num_objects': 792, 'num_object_clones': 0, 'num_object_copies': 3168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1612, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11276, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 345, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -820}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11459", 'reported_seq': 11636, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:53.438816+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:53.438816+0000', 'last_peered': '2024-09-10T20:29:53.438816+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:53.438816+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6829216, 'num_objects': 834, 'num_object_clones': 0, 'num_object_copies': 3336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11457, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1166}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11292", 'reported_seq': 11574, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:54.886678+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:54.886678+0000', 'last_peered': '2024-09-10T20:29:54.886678+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:54.886678+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6721264, 'num_objects': 821, 'num_object_clones': 0, 'num_object_copies': 3284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11289, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11319", 'reported_seq': 11623, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:54.802711+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:54.802711+0000', 'last_peered': '2024-09-10T20:29:54.802711+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:54.802711+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6928976, 'num_objects': 846, 'num_object_clones': 0, 'num_object_copies': 3384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11318, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11030", 'reported_seq': 11222, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:51.679639+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:29:51.679639+0000', 'last_peered': '2024-09-10T20:29:51.679639+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:29:51.679639+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6560336, 'num_objects': 801, 'num_object_clones': 0, 'num_object_copies': 3204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11029, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1102}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:29:57.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11584", 'reported_seq': 11798, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:51.455631+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:29:51.455631+0000', 'last_peered': '2024-09-10T20:29:51.455631+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:29:51.455631+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7083168, 'num_objects': 865, 'num_object_clones': 0, 'num_object_copies': 3460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11582, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:29:57.445 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:29:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:58.209 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18626/52560 objects degraded (35.438%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":686,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13140,"data_bytes":107612872,"bytes_used":2807074816,"bytes_avail":383739981824,"bytes_total":386547056640,"degraded_objects":18626,"degraded_total":52560,"degraded_ratio":0.35437595129375959,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:29:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:29:58.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:29:58.847 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:29:58.848 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:29:58.871 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-10T20:29:58.112320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105384648,"num_objects":12868,"num_object_clones":0,"num_object_copies":51472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12868,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141302,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160999,"ondisk_log_size":160999,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741372,"kb_used_data":210608,"kb_used_omap":14,"kb_used_meta":2530673,"kb_avail":374745988,"statfs":{"total":386547056640,"available":383739891712,"internally_reserved":0,"allocated":215662592,"data_stored":214734412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2591409718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,18,16,11,8,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18989056,"num_objects":-2318,"num_object_clones":0,"num_object_copies":-9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004207"},"pg_stats":[{"pgid":"1.f","version":"44'11322","reported_seq":11613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.573497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:54.573497+0000","last_peered":"2024-09-10T20:29:54.573497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:54.573497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":800,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11322,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11456","reported_seq":11597,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.307433+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:53.307433+0000","last_peered":"2024-09-10T20:29:53.307433+0000","last_clean":"2024-09-10T20:29:53.307433+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:53.307433+0000","last_undegraded":"2024-09-10T20:29:53.307433+0000","last_fullsized":"2024-09-10T20:29:53.307433+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6520832,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7897,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11399","reported_seq":11611,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:56.681921+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:29:56.681921+0000","last_peered":"2024-09-10T20:29:56.681921+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:29:56.681921+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6068840,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11398,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11547","reported_seq":15726,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.516890+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:53.516890+0000","last_peered":"2024-09-10T20:29:53.516890+0000","last_clean":"2024-09-10T20:29:53.516890+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:53.516890+0000","last_undegraded":"2024-09-10T20:29:53.516890+0000","last_fullsized":"2024-09-10T20:29:53.516890+0000","mapping_epoch":37,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7036928,"num_objects":859,"num_object_clones":0,"num_object_copies":3436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11336","reported_seq":11666,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.773873+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:54.773873+0000","last_peered":"2024-09-10T20:29:54.773873+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:54.773873+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6821024,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11334,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11294","reported_seq":11525,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.675381+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:54.675381+0000","last_peered":"2024-09-10T20:29:54.675381+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:54.675381+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1053}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11278","reported_seq":12058,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:53.404076+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:53.404076+0000","last_peered":"2024-09-10T20:29:53.404076+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:53.404076+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6485152,"num_objects":792,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11459","reported_seq":11636,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:53.438816+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:53.438816+0000","last_peered":"2024-09-10T20:29:53.438816+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:53.438816+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6829216,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11245","reported_seq":15226,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:54.654894+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:54.654894+0000","last_peered":"2024-09-10T20:29:54.654894+0000","last_clean":"2024-09-10T20:29:54.654894+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:54.654894+0000","last_undegraded":"2024-09-10T20:29:54.654894+0000","last_fullsized":"2024-09-10T20:29:54.654894+0000","mapping_epoch":33,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6191696,"num_objects":756,"num_object_clones":0,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11352","reported_seq":15332,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:56.683168+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:29:56.683168+0000","last_peered":"2024-09-10T20:29:56.683168+0000","last_clean":"2024-09-10T20:29:56.683168+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:29:56.683168+0000","last_undegraded":"2024-09-10T20:29:56.683168+0000","last_fullsized":"2024-09-10T20:29:56.683168+0000","mapping_epoch":39,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6471680,"num_objects":790,"num_object_clones":0,"num_object_copies":3160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11292","reported_seq":11574,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.886678+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:54.886678+0000","last_peered":"2024-09-10T20:29:54.886678+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:54.886678+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6721264,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11319","reported_seq":11623,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:54.802711+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:54.802711+0000","last_peered":"2024-09-10T20:29:54.802711+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:54.802711+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6928976,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11318,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1014}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11083","reported_seq":11275,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:56.648797+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:29:56.648797+0000","last_peered":"2024-09-10T20:29:56.648797+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:29:56.648797+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6126160,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1155}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11275","reported_seq":15335,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.315087+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:53.315087+0000","last_peered":"2024-09-10T20:29:53.315087+0000","last_clean":"2024-09-10T20:29:53.315087+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:53.315087+0000","last_undegraded":"2024-09-10T20:29:53.315087+0000","last_fullsized":"2024-09-10T20:29:53.315087+0000","mapping_epoch":43,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11654","reported_seq":11867,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:56.719331+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:29:56.719331+0000","last_peered":"2024-09-10T20:29:56.719331+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:29:56.719331+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6509728,"num_objects":795,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1113}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11288","reported_seq":15392,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:53.413645+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:53.413645+0000","last_peered":"2024-09-10T20:29:53.413645+0000","last_clean":"2024-09-10T20:29:53.413645+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:53.413645+0000","last_undegraded":"2024-09-10T20:29:53.413645+0000","last_fullsized":"2024-09-10T20:29:53.413645+0000","mapping_epoch":41,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840320,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":105384648,"num_objects":12868,"num_object_clones":0,"num_object_copies":51472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12868,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141302,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211738624,"data_stored":211738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160999,"ondisk_log_size":160999,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711856,"kb_used_data":52640,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93659984,"statfs":{"total":96636764160,"available":95907823616,"internally_reserved":0,"allocated":53903360,"data_stored":53671315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717968,"kb_used_data":50944,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93653872,"statfs":{"total":96636764160,"available":95901564928,"internally_reserved":0,"allocated":52166656,"data_stored":51934611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,9,1,0,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594592,"kb_used_data":53392,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93777248,"statfs":{"total":96636764160,"available":96027901952,"internally_reserved":0,"allocated":54673408,"data_stored":54441363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554168018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,4,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716956,"kb_used_data":53632,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93654884,"statfs":{"total":96636764160,"available":95902601216,"internally_reserved":0,"allocated":54919168,"data_stored":54687123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679210703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53972992,"data_stored":53972992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53891072,"data_stored":53891072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51134464,"data_stored":51134464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52740096,"data_stored":52740096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:01.873 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:02.485 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:02.485 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:02.512 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-10T20:30:02.113777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":96996040,"num_objects":11844,"num_object_clones":0,"num_object_copies":47376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142326,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745260,"kb_used_data":193664,"kb_used_omap":14,"kb_used_meta":2551473,"kb_avail":374742100,"statfs":{"total":386547056640,"available":383735910400,"internally_reserved":0,"allocated":198311936,"data_stored":197383756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2612708918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,9,8,6,13,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19226624,"num_objects":-2347,"num_object_clones":0,"num_object_copies":-9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004364"},"pg_stats":[{"pgid":"1.f","version":"44'11389","reported_seq":11680,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.897609+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:59.897609+0000","last_peered":"2024-09-10T20:29:59.897609+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:59.897609+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11511","reported_seq":11652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.588860+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:58.588860+0000","last_peered":"2024-09-10T20:29:58.588860+0000","last_clean":"2024-09-10T20:29:58.588860+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:58.588860+0000","last_undegraded":"2024-09-10T20:29:58.588860+0000","last_fullsized":"2024-09-10T20:29:58.588860+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6070272,"num_objects":741,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7952,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11461","reported_seq":11673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.699666+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:01.699666+0000","last_peered":"2024-09-10T20:30:01.699666+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:01.699666+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5560936,"num_objects":680,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11460,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11624","reported_seq":15803,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.602272+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:58.602272+0000","last_peered":"2024-09-10T20:29:58.602272+0000","last_clean":"2024-09-10T20:29:58.602272+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:58.602272+0000","last_undegraded":"2024-09-10T20:29:58.602272+0000","last_fullsized":"2024-09-10T20:29:58.602272+0000","mapping_epoch":37,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406144,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11395","reported_seq":11725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.926598+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:59.926598+0000","last_peered":"2024-09-10T20:29:59.926598+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:59.926598+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6337696,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11393,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11346","reported_seq":11577,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.784684+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:59.784684+0000","last_peered":"2024-09-10T20:29:59.784684+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:59.784684+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11348","reported_seq":12246,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:58.461581+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:58.461581+0000","last_peered":"2024-09-10T20:29:58.461581+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:58.461581+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5911712,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-831}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11516","reported_seq":11694,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:58.317404+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:58.317404+0000","last_peered":"2024-09-10T20:29:58.317404+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:58.317404+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6362272,"num_objects":777,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11305","reported_seq":15287,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:59.872517+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:59.872517+0000","last_peered":"2024-09-10T20:29:59.872517+0000","last_clean":"2024-09-10T20:29:59.872517+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:59.872517+0000","last_undegraded":"2024-09-10T20:29:59.872517+0000","last_fullsized":"2024-09-10T20:29:59.872517+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5700176,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11424","reported_seq":15404,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:01.789033+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:01.789033+0000","last_peered":"2024-09-10T20:30:01.789033+0000","last_clean":"2024-09-10T20:30:01.789033+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:01.789033+0000","last_undegraded":"2024-09-10T20:30:01.789033+0000","last_fullsized":"2024-09-10T20:30:01.789033+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5881856,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11353","reported_seq":11633,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.726880+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:59.726880+0000","last_peered":"2024-09-10T20:29:59.726880+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:59.726880+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6221552,"num_objects":760,"num_object_clones":0,"num_object_copies":3040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11350,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11391","reported_seq":11697,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.959595+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:59.959595+0000","last_peered":"2024-09-10T20:29:59.959595+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:59.959595+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6339152,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11140","reported_seq":11332,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.661315+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:01.661315+0000","last_peered":"2024-09-10T20:30:01.661315+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:01.661315+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5659216,"num_objects":691,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1212}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11337","reported_seq":15397,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.428231+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:58.428231+0000","last_peered":"2024-09-10T20:29:58.428231+0000","last_clean":"2024-09-10T20:29:58.428231+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:58.428231+0000","last_undegraded":"2024-09-10T20:29:58.428231+0000","last_fullsized":"2024-09-10T20:29:58.428231+0000","mapping_epoch":43,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6184960,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11728","reported_seq":11941,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.668896+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:01.668896+0000","last_peered":"2024-09-10T20:30:01.668896+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:01.668896+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5903520,"num_objects":721,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11355","reported_seq":15460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.355197+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:58.355197+0000","last_peered":"2024-09-10T20:29:58.355197+0000","last_clean":"2024-09-10T20:29:58.355197+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:58.355197+0000","last_undegraded":"2024-09-10T20:29:58.355197+0000","last_fullsized":"2024-09-10T20:29:58.355197+0000","mapping_epoch":41,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":768,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":96996040,"num_objects":11844,"num_object_clones":0,"num_object_copies":47376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142326,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194899968,"data_stored":194899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715376,"kb_used_data":47840,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93656464,"statfs":{"total":96636764160,"available":95904219136,"internally_reserved":0,"allocated":48988160,"data_stored":48756115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718144,"kb_used_data":46956,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93653696,"statfs":{"total":96636764160,"available":95901384704,"internally_reserved":0,"allocated":48082944,"data_stored":47850899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,5,0,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594668,"kb_used_data":49288,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93777172,"statfs":{"total":96636764160,"available":96027824128,"internally_reserved":0,"allocated":50470912,"data_stored":50238867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558427858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717072,"kb_used_data":49580,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93654768,"statfs":{"total":96636764160,"available":95902482432,"internally_reserved":0,"allocated":50769920,"data_stored":50537875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":683470543},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49770496,"data_stored":49770496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49700864,"data_stored":49700864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46907392,"data_stored":46907392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48521216,"data_stored":48521216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:02.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:03.127 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:03.127 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:03.152 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-10T20:30:02.113777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":96996040,"num_objects":11844,"num_object_clones":0,"num_object_copies":47376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142326,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745260,"kb_used_data":193664,"kb_used_omap":14,"kb_used_meta":2551473,"kb_avail":374742100,"statfs":{"total":386547056640,"available":383735910400,"internally_reserved":0,"allocated":198311936,"data_stored":197383756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2612708918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,9,8,6,13,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19226624,"num_objects":-2347,"num_object_clones":0,"num_object_copies":-9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004364"},"pg_stats":[{"pgid":"1.f","version":"44'11389","reported_seq":11680,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.897609+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:59.897609+0000","last_peered":"2024-09-10T20:29:59.897609+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:59.897609+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11511","reported_seq":11652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.588860+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:29:58.588860+0000","last_peered":"2024-09-10T20:29:58.588860+0000","last_clean":"2024-09-10T20:29:58.588860+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:29:58.588860+0000","last_undegraded":"2024-09-10T20:29:58.588860+0000","last_fullsized":"2024-09-10T20:29:58.588860+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6070272,"num_objects":741,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7952,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11461","reported_seq":11673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.699666+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:01.699666+0000","last_peered":"2024-09-10T20:30:01.699666+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:01.699666+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5560936,"num_objects":680,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11460,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11624","reported_seq":15803,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.602272+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:29:58.602272+0000","last_peered":"2024-09-10T20:29:58.602272+0000","last_clean":"2024-09-10T20:29:58.602272+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:29:58.602272+0000","last_undegraded":"2024-09-10T20:29:58.602272+0000","last_fullsized":"2024-09-10T20:29:58.602272+0000","mapping_epoch":37,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406144,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11395","reported_seq":11725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.926598+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:59.926598+0000","last_peered":"2024-09-10T20:29:59.926598+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:59.926598+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6337696,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11393,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11346","reported_seq":11577,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.784684+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:59.784684+0000","last_peered":"2024-09-10T20:29:59.784684+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:59.784684+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11348","reported_seq":12246,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:58.461581+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:29:58.461581+0000","last_peered":"2024-09-10T20:29:58.461581+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:29:58.461581+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5911712,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-831}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11516","reported_seq":11694,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:58.317404+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:29:58.317404+0000","last_peered":"2024-09-10T20:29:58.317404+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:29:58.317404+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6362272,"num_objects":777,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11305","reported_seq":15287,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:59.872517+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:59.872517+0000","last_peered":"2024-09-10T20:29:59.872517+0000","last_clean":"2024-09-10T20:29:59.872517+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:59.872517+0000","last_undegraded":"2024-09-10T20:29:59.872517+0000","last_fullsized":"2024-09-10T20:29:59.872517+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5700176,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11424","reported_seq":15404,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:01.789033+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:01.789033+0000","last_peered":"2024-09-10T20:30:01.789033+0000","last_clean":"2024-09-10T20:30:01.789033+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:01.789033+0000","last_undegraded":"2024-09-10T20:30:01.789033+0000","last_fullsized":"2024-09-10T20:30:01.789033+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5881856,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11353","reported_seq":11633,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.726880+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:59.726880+0000","last_peered":"2024-09-10T20:29:59.726880+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:59.726880+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6221552,"num_objects":760,"num_object_clones":0,"num_object_copies":3040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11350,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11391","reported_seq":11697,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.959595+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:59.959595+0000","last_peered":"2024-09-10T20:29:59.959595+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:59.959595+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6339152,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11140","reported_seq":11332,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.661315+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:01.661315+0000","last_peered":"2024-09-10T20:30:01.661315+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:01.661315+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5659216,"num_objects":691,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1212}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11337","reported_seq":15397,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.428231+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:29:58.428231+0000","last_peered":"2024-09-10T20:29:58.428231+0000","last_clean":"2024-09-10T20:29:58.428231+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:29:58.428231+0000","last_undegraded":"2024-09-10T20:29:58.428231+0000","last_fullsized":"2024-09-10T20:29:58.428231+0000","mapping_epoch":43,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6184960,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11728","reported_seq":11941,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.668896+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:01.668896+0000","last_peered":"2024-09-10T20:30:01.668896+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:01.668896+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5903520,"num_objects":721,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11355","reported_seq":15460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:58.355197+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:29:58.355197+0000","last_peered":"2024-09-10T20:29:58.355197+0000","last_clean":"2024-09-10T20:29:58.355197+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:29:58.355197+0000","last_undegraded":"2024-09-10T20:29:58.355197+0000","last_fullsized":"2024-09-10T20:29:58.355197+0000","mapping_epoch":41,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":768,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":96996040,"num_objects":11844,"num_object_clones":0,"num_object_copies":47376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142326,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194899968,"data_stored":194899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715376,"kb_used_data":47840,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93656464,"statfs":{"total":96636764160,"available":95904219136,"internally_reserved":0,"allocated":48988160,"data_stored":48756115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718144,"kb_used_data":46956,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93653696,"statfs":{"total":96636764160,"available":95901384704,"internally_reserved":0,"allocated":48082944,"data_stored":47850899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,5,0,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594668,"kb_used_data":49288,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93777172,"statfs":{"total":96636764160,"available":96027824128,"internally_reserved":0,"allocated":50470912,"data_stored":50238867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558427858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717072,"kb_used_data":49580,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93654768,"statfs":{"total":96636764160,"available":95902482432,"internally_reserved":0,"allocated":50769920,"data_stored":50537875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":683470543},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49770496,"data_stored":49770496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49700864,"data_stored":49700864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46907392,"data_stored":46907392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48521216,"data_stored":48521216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:03.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:03.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11389", 'reported_seq': 11680, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:59.897609+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:29:59.897609+0000', 'last_peered': '2024-09-10T20:29:59.897609+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:29:59.897609+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6004736, 'num_objects': 733, 'num_object_clones': 0, 'num_object_copies': 2932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11389, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1122}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:03.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:03.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11461", 'reported_seq': 11673, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:01.699666+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:01.699666+0000', 'last_peered': '2024-09-10T20:30:01.699666+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:01.699666+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5560936, 'num_objects': 680, 'num_object_clones': 0, 'num_object_copies': 2720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 680, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11460, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1228}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:03.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:03.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11395", 'reported_seq': 11725, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:59.926598+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:29:59.926598+0000', 'last_peered': '2024-09-10T20:29:59.926598+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:29:59.926598+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6337696, 'num_objects': 774, 'num_object_clones': 0, 'num_object_copies': 3096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11393, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1081}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:03.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11346", 'reported_seq': 11577, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:59.784684+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:29:59.784684+0000', 'last_peered': '2024-09-10T20:29:59.784684+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:29:59.784684+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6160384, 'num_objects': 752, 'num_object_clones': 0, 'num_object_copies': 3008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11346, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1105}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11348", 'reported_seq': 12246, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:58.461581+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:29:58.461581+0000', 'last_peered': '2024-09-10T20:29:58.461581+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:29:58.461581+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5911712, 'num_objects': 722, 'num_object_clones': 0, 'num_object_copies': 2888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11346, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 404, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -831}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11516", 'reported_seq': 11694, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:58.317404+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:29:58.317404+0000', 'last_peered': '2024-09-10T20:29:58.317404+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:29:58.317404+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6362272, 'num_objects': 777, 'num_object_clones': 0, 'num_object_copies': 3108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11514, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1223}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11353", 'reported_seq': 11633, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:59.726880+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:29:59.726880+0000', 'last_peered': '2024-09-10T20:29:59.726880+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:29:59.726880+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6221552, 'num_objects': 760, 'num_object_clones': 0, 'num_object_copies': 3040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11350, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1096}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:03.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11391", 'reported_seq': 11697, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:29:59.959595+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:29:59.959595+0000', 'last_peered': '2024-09-10T20:29:59.959595+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:29:59.959595+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6339152, 'num_objects': 774, 'num_object_clones': 0, 'num_object_copies': 3096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11390, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:03.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:03.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11140", 'reported_seq': 11332, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:01.661315+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:01.661315+0000', 'last_peered': '2024-09-10T20:30:01.661315+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:01.661315+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5659216, 'num_objects': 691, 'num_object_clones': 0, 'num_object_copies': 2764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11139, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1212}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:03.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:03.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11728", 'reported_seq': 11941, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:01.668896+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:01.668896+0000', 'last_peered': '2024-09-10T20:30:01.668896+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:01.668896+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5903520, 'num_objects': 721, 'num_object_clones': 0, 'num_object_copies': 2884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11726, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1187}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:03.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:03.908 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18567/47376 objects degraded (39.191%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":692,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11844,"data_bytes":96996040,"bytes_used":2811146240,"bytes_avail":383735910400,"bytes_total":386547056640,"degraded_objects":18567,"degraded_total":47376,"degraded_ratio":0.39190729483282671,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-10T20:28:04.067296+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:03.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:04.582 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:04.583 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:04.609 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-10T20:30:04.114388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93842120,"num_objects":11459,"num_object_clones":0,"num_object_copies":45836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142711,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160908,"ondisk_log_size":160908,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745472,"kb_used_data":185588,"kb_used_omap":14,"kb_used_meta":2559793,"kb_avail":374741888,"statfs":{"total":386547056640,"available":383735693312,"internally_reserved":0,"allocated":190042112,"data_stored":189113932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2621228598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,17,6,2,12,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20045824,"num_objects":-2447,"num_object_clones":0,"num_object_copies":-9788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2447,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004149"},"pg_stats":[{"pgid":"1.f","version":"44'11389","reported_seq":11680,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.897609+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:29:59.897609+0000","last_peered":"2024-09-10T20:29:59.897609+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:29:59.897609+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11580","reported_seq":11720,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.702939+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:03.702939+0000","last_peered":"2024-09-10T20:30:03.702939+0000","last_clean":"2024-09-10T20:30:03.702939+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:03.702939+0000","last_undegraded":"2024-09-10T20:30:03.702939+0000","last_fullsized":"2024-09-10T20:30:03.702939+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":672,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8021,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11461","reported_seq":11673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.699666+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:01.699666+0000","last_peered":"2024-09-10T20:30:01.699666+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:01.699666+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5560936,"num_objects":680,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11460,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11688","reported_seq":15867,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.251788+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:03.251788+0000","last_peered":"2024-09-10T20:30:03.251788+0000","last_clean":"2024-09-10T20:30:03.251788+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:03.251788+0000","last_undegraded":"2024-09-10T20:30:03.251788+0000","last_fullsized":"2024-09-10T20:30:03.251788+0000","mapping_epoch":37,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5881856,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5948,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11395","reported_seq":11725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.926598+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:29:59.926598+0000","last_peered":"2024-09-10T20:29:59.926598+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:29:59.926598+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6337696,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11393,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11346","reported_seq":11577,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.784684+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:29:59.784684+0000","last_peered":"2024-09-10T20:29:59.784684+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:29:59.784684+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11408","reported_seq":12448,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:03.545866+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:03.545866+0000","last_peered":"2024-09-10T20:30:03.545866+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:03.545866+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5420192,"num_objects":662,"num_object_clones":0,"num_object_copies":2648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11573","reported_seq":11751,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:03.401662+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:03.401662+0000","last_peered":"2024-09-10T20:30:03.401662+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:03.401662+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5895328,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11305","reported_seq":15287,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:29:59.872517+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:29:59.872517+0000","last_peered":"2024-09-10T20:29:59.872517+0000","last_clean":"2024-09-10T20:29:59.872517+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:29:59.872517+0000","last_undegraded":"2024-09-10T20:29:59.872517+0000","last_fullsized":"2024-09-10T20:29:59.872517+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5700176,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11424","reported_seq":15404,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:01.789033+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:01.789033+0000","last_peered":"2024-09-10T20:30:01.789033+0000","last_clean":"2024-09-10T20:30:01.789033+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:01.789033+0000","last_undegraded":"2024-09-10T20:30:01.789033+0000","last_fullsized":"2024-09-10T20:30:01.789033+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5881856,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11353","reported_seq":11633,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.726880+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:29:59.726880+0000","last_peered":"2024-09-10T20:29:59.726880+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:29:59.726880+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6221552,"num_objects":760,"num_object_clones":0,"num_object_copies":3040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11350,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11391","reported_seq":11697,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:29:59.959595+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:29:59.959595+0000","last_peered":"2024-09-10T20:29:59.959595+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:29:59.959595+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6339152,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11140","reported_seq":11332,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.661315+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:01.661315+0000","last_peered":"2024-09-10T20:30:01.661315+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:01.661315+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5659216,"num_objects":691,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1212}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11399","reported_seq":15459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.659971+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:03.659971+0000","last_peered":"2024-09-10T20:30:03.659971+0000","last_clean":"2024-09-10T20:30:03.659971+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:03.659971+0000","last_undegraded":"2024-09-10T20:30:03.659971+0000","last_fullsized":"2024-09-10T20:30:03.659971+0000","mapping_epoch":43,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5677056,"num_objects":693,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11728","reported_seq":11941,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:01.668896+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:01.668896+0000","last_peered":"2024-09-10T20:30:01.668896+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:01.668896+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5903520,"num_objects":721,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11428","reported_seq":15533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.629229+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:03.629229+0000","last_peered":"2024-09-10T20:30:03.629229+0000","last_clean":"2024-09-10T20:30:03.629229+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:03.629229+0000","last_undegraded":"2024-09-10T20:30:03.629229+0000","last_fullsized":"2024-09-10T20:30:03.629229+0000","mapping_epoch":41,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5693440,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":93842120,"num_objects":11459,"num_object_clones":0,"num_object_copies":45836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142711,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186433536,"data_stored":186433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160908,"ondisk_log_size":160908,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715376,"kb_used_data":47840,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93656464,"statfs":{"total":96636764160,"available":95904219136,"internally_reserved":0,"allocated":48988160,"data_stored":48756115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718144,"kb_used_data":46956,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93653696,"statfs":{"total":96636764160,"available":95901384704,"internally_reserved":0,"allocated":48082944,"data_stored":47850899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,5,0,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018999,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594784,"kb_used_data":45256,"kb_used_omap":4,"kb_used_meta":549499,"kb_avail":93777056,"statfs":{"total":96636764160,"available":96027705344,"internally_reserved":0,"allocated":46342144,"data_stored":46110099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":562687698},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215237,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717168,"kb_used_data":45536,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93654672,"statfs":{"total":96636764160,"available":95902384128,"internally_reserved":0,"allocated":46628864,"data_stored":46396819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,0,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45551616,"data_stored":45551616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45453312,"data_stored":45453312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46907392,"data_stored":46907392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48521216,"data_stored":48521216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:07.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:08.228 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:08.229 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:08.264 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-10T20:30:08.119739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88509128,"num_objects":10808,"num_object_clones":0,"num_object_copies":43232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143362,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725396,"kb_used_data":177984,"kb_used_omap":14,"kb_used_meta":2547313,"kb_avail":374761964,"statfs":{"total":386547056640,"available":383756251136,"internally_reserved":0,"allocated":182255616,"data_stored":181327436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2608449078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,11,11,1,13,11,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19103744,"num_objects":-2332,"num_object_clones":0,"num_object_copies":-9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2332,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008235"},"pg_stats":[{"pgid":"1.f","version":"44'11448","reported_seq":11739,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:04.924694+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:04.924694+0000","last_peered":"2024-09-10T20:30:04.924694+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:04.924694+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5521408,"num_objects":674,"num_object_clones":0,"num_object_copies":2696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11580","reported_seq":11720,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.702939+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:03.702939+0000","last_peered":"2024-09-10T20:30:03.702939+0000","last_clean":"2024-09-10T20:30:03.702939+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:03.702939+0000","last_undegraded":"2024-09-10T20:30:03.702939+0000","last_fullsized":"2024-09-10T20:30:03.702939+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":672,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8021,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11523","reported_seq":11735,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.657328+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:06.657328+0000","last_peered":"2024-09-10T20:30:06.657328+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:06.657328+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5053032,"num_objects":618,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11522,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11688","reported_seq":15867,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.251788+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:03.251788+0000","last_peered":"2024-09-10T20:30:03.251788+0000","last_clean":"2024-09-10T20:30:03.251788+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:03.251788+0000","last_undegraded":"2024-09-10T20:30:03.251788+0000","last_fullsized":"2024-09-10T20:30:03.251788+0000","mapping_epoch":37,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5881856,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5948,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11445","reported_seq":11775,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.040439+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:05.040439+0000","last_peered":"2024-09-10T20:30:05.040439+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:05.040439+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5928096,"num_objects":724,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11412","reported_seq":11641,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:04.980638+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:04.980638+0000","last_peered":"2024-09-10T20:30:04.980638+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:04.980638+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5619712,"num_objects":686,"num_object_clones":0,"num_object_copies":2744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11412,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11408","reported_seq":12448,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:03.545866+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:03.545866+0000","last_peered":"2024-09-10T20:30:03.545866+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:03.545866+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5420192,"num_objects":662,"num_object_clones":0,"num_object_copies":2648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11573","reported_seq":11751,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:03.401662+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:03.401662+0000","last_peered":"2024-09-10T20:30:03.401662+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:03.401662+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5895328,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11379","reported_seq":15361,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:04.980321+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:04.980321+0000","last_peered":"2024-09-10T20:30:04.980321+0000","last_clean":"2024-09-10T20:30:04.980321+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:04.980321+0000","last_undegraded":"2024-09-10T20:30:04.980321+0000","last_fullsized":"2024-09-10T20:30:04.980321+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093968,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11484","reported_seq":15464,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:06.747506+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:06.747506+0000","last_peered":"2024-09-10T20:30:06.747506+0000","last_clean":"2024-09-10T20:30:06.747506+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:06.747506+0000","last_undegraded":"2024-09-10T20:30:06.747506+0000","last_fullsized":"2024-09-10T20:30:06.747506+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5390336,"num_objects":658,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11428","reported_seq":11710,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.123696+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:05.123696+0000","last_peered":"2024-09-10T20:30:05.123696+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:05.123696+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5607152,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11425,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11455","reported_seq":11759,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.126054+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:05.126054+0000","last_peered":"2024-09-10T20:30:05.126054+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:05.126054+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5814864,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11209","reported_seq":11402,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.828487+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:06.828487+0000","last_peered":"2024-09-10T20:30:06.828487+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:06.828487+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093968,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11399","reported_seq":15459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.659971+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:03.659971+0000","last_peered":"2024-09-10T20:30:03.659971+0000","last_clean":"2024-09-10T20:30:03.659971+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:03.659971+0000","last_undegraded":"2024-09-10T20:30:03.659971+0000","last_fullsized":"2024-09-10T20:30:03.659971+0000","mapping_epoch":43,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5677056,"num_objects":693,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11800","reported_seq":12015,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.708019+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:06.708019+0000","last_peered":"2024-09-10T20:30:06.708019+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:06.708019+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5313696,"num_objects":649,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11798,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11428","reported_seq":15533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.629229+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:03.629229+0000","last_peered":"2024-09-10T20:30:03.629229+0000","last_clean":"2024-09-10T20:30:03.629229+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:03.629229+0000","last_undegraded":"2024-09-10T20:30:03.629229+0000","last_fullsized":"2024-09-10T20:30:03.629229+0000","mapping_epoch":41,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5693440,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":88509128,"num_objects":10808,"num_object_clones":0,"num_object_copies":43232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143362,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":177979392,"data_stored":177979392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711920,"kb_used_data":44384,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93659920,"statfs":{"total":96636764160,"available":95907758080,"internally_reserved":0,"allocated":45449216,"data_stored":45217171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,0,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701524,"kb_used_data":42808,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93670316,"statfs":{"total":96636764160,"available":95918403584,"internally_reserved":0,"allocated":43835392,"data_stored":43603347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,1,2,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018999,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594784,"kb_used_data":45256,"kb_used_omap":4,"kb_used_meta":549499,"kb_avail":93777056,"statfs":{"total":96636764160,"available":96027705344,"internally_reserved":0,"allocated":46342144,"data_stored":46110099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":562687698},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215237,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717168,"kb_used_data":45536,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93654672,"statfs":{"total":96636764160,"available":95902384128,"internally_reserved":0,"allocated":46628864,"data_stored":46396819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,0,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45551616,"data_stored":45551616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45453312,"data_stored":45453312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42733568,"data_stored":42733568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44240896,"data_stored":44240896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:08.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:08.869 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:08.869 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:08.889 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-10T20:30:08.119739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88509128,"num_objects":10808,"num_object_clones":0,"num_object_copies":43232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143362,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725396,"kb_used_data":177984,"kb_used_omap":14,"kb_used_meta":2547313,"kb_avail":374761964,"statfs":{"total":386547056640,"available":383756251136,"internally_reserved":0,"allocated":182255616,"data_stored":181327436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2608449078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,11,11,1,13,11,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19103744,"num_objects":-2332,"num_object_clones":0,"num_object_copies":-9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2332,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008235"},"pg_stats":[{"pgid":"1.f","version":"44'11448","reported_seq":11739,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:04.924694+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:04.924694+0000","last_peered":"2024-09-10T20:30:04.924694+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:04.924694+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5521408,"num_objects":674,"num_object_clones":0,"num_object_copies":2696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11580","reported_seq":11720,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.702939+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:03.702939+0000","last_peered":"2024-09-10T20:30:03.702939+0000","last_clean":"2024-09-10T20:30:03.702939+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:03.702939+0000","last_undegraded":"2024-09-10T20:30:03.702939+0000","last_fullsized":"2024-09-10T20:30:03.702939+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":672,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8021,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11523","reported_seq":11735,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.657328+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:06.657328+0000","last_peered":"2024-09-10T20:30:06.657328+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:06.657328+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5053032,"num_objects":618,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11522,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11688","reported_seq":15867,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.251788+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:03.251788+0000","last_peered":"2024-09-10T20:30:03.251788+0000","last_clean":"2024-09-10T20:30:03.251788+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:03.251788+0000","last_undegraded":"2024-09-10T20:30:03.251788+0000","last_fullsized":"2024-09-10T20:30:03.251788+0000","mapping_epoch":37,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5881856,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5948,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11445","reported_seq":11775,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.040439+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:05.040439+0000","last_peered":"2024-09-10T20:30:05.040439+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:05.040439+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5928096,"num_objects":724,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11412","reported_seq":11641,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:04.980638+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:04.980638+0000","last_peered":"2024-09-10T20:30:04.980638+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:04.980638+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5619712,"num_objects":686,"num_object_clones":0,"num_object_copies":2744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11412,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11408","reported_seq":12448,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:03.545866+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:03.545866+0000","last_peered":"2024-09-10T20:30:03.545866+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:03.545866+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5420192,"num_objects":662,"num_object_clones":0,"num_object_copies":2648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11573","reported_seq":11751,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:03.401662+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:03.401662+0000","last_peered":"2024-09-10T20:30:03.401662+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:03.401662+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5895328,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11379","reported_seq":15361,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:04.980321+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:04.980321+0000","last_peered":"2024-09-10T20:30:04.980321+0000","last_clean":"2024-09-10T20:30:04.980321+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:04.980321+0000","last_undegraded":"2024-09-10T20:30:04.980321+0000","last_fullsized":"2024-09-10T20:30:04.980321+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093968,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11484","reported_seq":15464,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:06.747506+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:06.747506+0000","last_peered":"2024-09-10T20:30:06.747506+0000","last_clean":"2024-09-10T20:30:06.747506+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:06.747506+0000","last_undegraded":"2024-09-10T20:30:06.747506+0000","last_fullsized":"2024-09-10T20:30:06.747506+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5390336,"num_objects":658,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11428","reported_seq":11710,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.123696+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:05.123696+0000","last_peered":"2024-09-10T20:30:05.123696+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:05.123696+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5607152,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11425,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11455","reported_seq":11759,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.126054+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:05.126054+0000","last_peered":"2024-09-10T20:30:05.126054+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:05.126054+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5814864,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11209","reported_seq":11402,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.828487+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:06.828487+0000","last_peered":"2024-09-10T20:30:06.828487+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:06.828487+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093968,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11399","reported_seq":15459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.659971+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:03.659971+0000","last_peered":"2024-09-10T20:30:03.659971+0000","last_clean":"2024-09-10T20:30:03.659971+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:03.659971+0000","last_undegraded":"2024-09-10T20:30:03.659971+0000","last_fullsized":"2024-09-10T20:30:03.659971+0000","mapping_epoch":43,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5677056,"num_objects":693,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11800","reported_seq":12015,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.708019+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:06.708019+0000","last_peered":"2024-09-10T20:30:06.708019+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:06.708019+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5313696,"num_objects":649,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11798,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11428","reported_seq":15533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:03.629229+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:03.629229+0000","last_peered":"2024-09-10T20:30:03.629229+0000","last_clean":"2024-09-10T20:30:03.629229+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:03.629229+0000","last_undegraded":"2024-09-10T20:30:03.629229+0000","last_fullsized":"2024-09-10T20:30:03.629229+0000","mapping_epoch":41,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5693440,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":88509128,"num_objects":10808,"num_object_clones":0,"num_object_copies":43232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143362,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":177979392,"data_stored":177979392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711920,"kb_used_data":44384,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93659920,"statfs":{"total":96636764160,"available":95907758080,"internally_reserved":0,"allocated":45449216,"data_stored":45217171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,0,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701524,"kb_used_data":42808,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93670316,"statfs":{"total":96636764160,"available":95918403584,"internally_reserved":0,"allocated":43835392,"data_stored":43603347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,1,2,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018999,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594784,"kb_used_data":45256,"kb_used_omap":4,"kb_used_meta":549499,"kb_avail":93777056,"statfs":{"total":96636764160,"available":96027705344,"internally_reserved":0,"allocated":46342144,"data_stored":46110099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":562687698},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215237,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717168,"kb_used_data":45536,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93654672,"statfs":{"total":96636764160,"available":95902384128,"internally_reserved":0,"allocated":46628864,"data_stored":46396819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,0,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45551616,"data_stored":45551616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45453312,"data_stored":45453312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42733568,"data_stored":42733568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44240896,"data_stored":44240896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11448", 'reported_seq': 11739, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:04.924694+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:04.924694+0000', 'last_peered': '2024-09-10T20:30:04.924694+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:04.924694+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5521408, 'num_objects': 674, 'num_object_clones': 0, 'num_object_copies': 2696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11448, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1181}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11523", 'reported_seq': 11735, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:06.657328+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:06.657328+0000', 'last_peered': '2024-09-10T20:30:06.657328+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:06.657328+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5053032, 'num_objects': 618, 'num_object_clones': 0, 'num_object_copies': 2472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 618, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11522, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1290}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11445", 'reported_seq': 11775, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:05.040439+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:05.040439+0000', 'last_peered': '2024-09-10T20:30:05.040439+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:05.040439+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5928096, 'num_objects': 724, 'num_object_clones': 0, 'num_object_copies': 2896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11443, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1131}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11412", 'reported_seq': 11641, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:04.980638+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:04.980638+0000', 'last_peered': '2024-09-10T20:30:04.980638+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:04.980638+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5619712, 'num_objects': 686, 'num_object_clones': 0, 'num_object_copies': 2744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11412, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1171}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:08.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11408", 'reported_seq': 12448, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:03.545866+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:03.545866+0000', 'last_peered': '2024-09-10T20:30:03.545866+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:03.545866+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5420192, 'num_objects': 662, 'num_object_clones': 0, 'num_object_copies': 2648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1482, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11406, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 475, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -820}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11573", 'reported_seq': 11751, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:03.401662+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:03.401662+0000', 'last_peered': '2024-09-10T20:30:03.401662+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:03.401662+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5895328, 'num_objects': 720, 'num_object_clones': 0, 'num_object_copies': 2880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11571, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11428", 'reported_seq': 11710, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:05.123696+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:05.123696+0000', 'last_peered': '2024-09-10T20:30:05.123696+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:05.123696+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5607152, 'num_objects': 685, 'num_object_clones': 0, 'num_object_copies': 2740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11425, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1171}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11455", 'reported_seq': 11759, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:05.126054+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:05.126054+0000', 'last_peered': '2024-09-10T20:30:05.126054+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:05.126054+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5814864, 'num_objects': 710, 'num_object_clones': 0, 'num_object_copies': 2840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11454, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11209", 'reported_seq': 11402, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:06.828487+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:06.828487+0000', 'last_peered': '2024-09-10T20:30:06.828487+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:06.828487+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5093968, 'num_objects': 622, 'num_object_clones': 0, 'num_object_copies': 2488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11208, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:08.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11800", 'reported_seq': 12015, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:06.708019+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:06.708019+0000', 'last_peered': '2024-09-10T20:30:06.708019+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:06.708019+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5313696, 'num_objects': 649, 'num_object_clones': 0, 'num_object_copies': 2596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11798, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:08.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:09.708 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:09.708 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18496/43232 objects degraded (42.783%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":698,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10808,"data_bytes":88509128,"bytes_used":2790805504,"bytes_avail":383756251136,"bytes_total":386547056640,"degraded_objects":18496,"degraded_total":43232,"degraded_ratio":0.42783123612139157,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:09.750 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:09.750 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:10.348 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:10.369 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-10T20:30:10.120673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85224136,"num_objects":10407,"num_object_clones":0,"num_object_copies":41628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10407,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143763,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160860,"ondisk_log_size":160860,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691316,"kb_used_data":168872,"kb_used_omap":14,"kb_used_meta":2522353,"kb_avail":374796044,"statfs":{"total":386547056640,"available":383791149056,"internally_reserved":0,"allocated":172924928,"data_stored":171996748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2582890038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,4,2,13,9,23,10,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20160512,"num_objects":-2461,"num_object_clones":0,"num_object_copies":-9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008353"},"pg_stats":[{"pgid":"1.f","version":"44'11448","reported_seq":11739,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:04.924694+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:04.924694+0000","last_peered":"2024-09-10T20:30:04.924694+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:04.924694+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5521408,"num_objects":674,"num_object_clones":0,"num_object_copies":2696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11664","reported_seq":11805,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.961346+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:08.961346+0000","last_peered":"2024-09-10T20:30:08.961346+0000","last_clean":"2024-09-10T20:30:08.961346+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:08.961346+0000","last_undegraded":"2024-09-10T20:30:08.961346+0000","last_fullsized":"2024-09-10T20:30:08.961346+0000","mapping_epoch":35,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4816896,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8105,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11523","reported_seq":11735,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.657328+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:06.657328+0000","last_peered":"2024-09-10T20:30:06.657328+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:06.657328+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5053032,"num_objects":618,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11522,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11755","reported_seq":15934,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.929666+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:08.929666+0000","last_peered":"2024-09-10T20:30:08.929666+0000","last_clean":"2024-09-10T20:30:08.929666+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:08.929666+0000","last_undegraded":"2024-09-10T20:30:08.929666+0000","last_fullsized":"2024-09-10T20:30:08.929666+0000","mapping_epoch":37,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5332992,"num_objects":651,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11445","reported_seq":11775,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.040439+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:05.040439+0000","last_peered":"2024-09-10T20:30:05.040439+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:05.040439+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5928096,"num_objects":724,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11412","reported_seq":11641,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:04.980638+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:04.980638+0000","last_peered":"2024-09-10T20:30:04.980638+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:04.980638+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5619712,"num_objects":686,"num_object_clones":0,"num_object_copies":2744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11412,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11463","reported_seq":12611,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:08.530818+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:08.530818+0000","last_peered":"2024-09-10T20:30:08.530818+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:08.530818+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969632,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":529,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11637","reported_seq":11815,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:08.253335+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:08.253335+0000","last_peered":"2024-09-10T20:30:08.253335+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:08.253335+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5371040,"num_objects":656,"num_object_clones":0,"num_object_copies":2624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11379","reported_seq":15361,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:04.980321+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:04.980321+0000","last_peered":"2024-09-10T20:30:04.980321+0000","last_clean":"2024-09-10T20:30:04.980321+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:04.980321+0000","last_undegraded":"2024-09-10T20:30:04.980321+0000","last_fullsized":"2024-09-10T20:30:04.980321+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093968,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11484","reported_seq":15464,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:06.747506+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:06.747506+0000","last_peered":"2024-09-10T20:30:06.747506+0000","last_clean":"2024-09-10T20:30:06.747506+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:06.747506+0000","last_undegraded":"2024-09-10T20:30:06.747506+0000","last_fullsized":"2024-09-10T20:30:06.747506+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5390336,"num_objects":658,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11428","reported_seq":11710,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.123696+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:05.123696+0000","last_peered":"2024-09-10T20:30:05.123696+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:05.123696+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5607152,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11425,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11455","reported_seq":11759,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:05.126054+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:05.126054+0000","last_peered":"2024-09-10T20:30:05.126054+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:05.126054+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5814864,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11209","reported_seq":11402,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.828487+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:06.828487+0000","last_peered":"2024-09-10T20:30:06.828487+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:06.828487+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093968,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11465","reported_seq":15525,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.861648+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:08.861648+0000","last_peered":"2024-09-10T20:30:08.861648+0000","last_clean":"2024-09-10T20:30:08.861648+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:08.861648+0000","last_undegraded":"2024-09-10T20:30:08.861648+0000","last_fullsized":"2024-09-10T20:30:08.861648+0000","mapping_epoch":43,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5136384,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11800","reported_seq":12015,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:06.708019+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:06.708019+0000","last_peered":"2024-09-10T20:30:06.708019+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:06.708019+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5313696,"num_objects":649,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11798,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11493","reported_seq":15598,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.825681+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:08.825681+0000","last_peered":"2024-09-10T20:30:08.825681+0000","last_clean":"2024-09-10T20:30:08.825681+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:08.825681+0000","last_undegraded":"2024-09-10T20:30:08.825681+0000","last_fullsized":"2024-09-10T20:30:08.825681+0000","mapping_epoch":41,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5160960,"num_objects":630,"num_object_clones":0,"num_object_copies":2520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2366,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":85224136,"num_objects":10407,"num_object_clones":0,"num_object_copies":41628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10407,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143763,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169156608,"data_stored":169156608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160860,"ondisk_log_size":160860,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711920,"kb_used_data":44384,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93659920,"statfs":{"total":96636764160,"available":95907758080,"internally_reserved":0,"allocated":45449216,"data_stored":45217171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,0,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701524,"kb_used_data":42808,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93670316,"statfs":{"total":96636764160,"available":95918403584,"internally_reserved":0,"allocated":43835392,"data_stored":43603347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,1,2,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578076,"kb_used_data":41036,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93793764,"statfs":{"total":96636764160,"available":96044814336,"internally_reserved":0,"allocated":42020864,"data_stored":41788819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,2,8,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215238,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699796,"kb_used_data":40644,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93672044,"statfs":{"total":96636764160,"available":95920173056,"internally_reserved":0,"allocated":41619456,"data_stored":41387411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,7,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41230336,"data_stored":41230336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40951808,"data_stored":40951808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42733568,"data_stored":42733568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44240896,"data_stored":44240896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:13.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:14.036 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:14.036 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:14.069 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-10T20:30:12.121434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80177864,"num_objects":9791,"num_object_clones":0,"num_object_copies":39164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144379,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691124,"kb_used_data":160360,"kb_used_omap":14,"kb_used_meta":2530673,"kb_avail":374796236,"statfs":{"total":386547056640,"available":383791345664,"internally_reserved":0,"allocated":164208640,"data_stored":163280460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2591409718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,11,15,19,15,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18989056,"num_objects":-2318,"num_object_clones":0,"num_object_copies":-9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008316"},"pg_stats":[{"pgid":"1.f","version":"44'11520","reported_seq":11811,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:09.795086+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:09.795086+0000","last_peered":"2024-09-10T20:30:09.795086+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:09.795086+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4931584,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11664","reported_seq":11805,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.961346+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:08.961346+0000","last_peered":"2024-09-10T20:30:08.961346+0000","last_clean":"2024-09-10T20:30:08.961346+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:08.961346+0000","last_undegraded":"2024-09-10T20:30:08.961346+0000","last_fullsized":"2024-09-10T20:30:08.961346+0000","mapping_epoch":35,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4816896,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8105,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11570","reported_seq":11781,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.642821+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:11.642821+0000","last_peered":"2024-09-10T20:30:11.642821+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:11.642821+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4668008,"num_objects":571,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11569,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11755","reported_seq":15934,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.929666+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:08.929666+0000","last_peered":"2024-09-10T20:30:08.929666+0000","last_clean":"2024-09-10T20:30:08.929666+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:08.929666+0000","last_undegraded":"2024-09-10T20:30:08.929666+0000","last_fullsized":"2024-09-10T20:30:08.929666+0000","mapping_epoch":37,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5332992,"num_objects":651,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11506","reported_seq":11836,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:10.026231+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:10.026231+0000","last_peered":"2024-09-10T20:30:10.026231+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:10.026231+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428384,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11477","reported_seq":11708,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:10.132824+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:10.132824+0000","last_peered":"2024-09-10T20:30:10.132824+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:10.132824+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5087232,"num_objects":621,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11463","reported_seq":12611,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:08.530818+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:08.530818+0000","last_peered":"2024-09-10T20:30:08.530818+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:08.530818+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969632,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":529,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11637","reported_seq":11815,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:08.253335+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:08.253335+0000","last_peered":"2024-09-10T20:30:08.253335+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:08.253335+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5371040,"num_objects":656,"num_object_clones":0,"num_object_copies":2624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11445","reported_seq":15427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:10.133864+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:10.133864+0000","last_peered":"2024-09-10T20:30:10.133864+0000","last_clean":"2024-09-10T20:30:10.133864+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:10.133864+0000","last_undegraded":"2024-09-10T20:30:10.133864+0000","last_fullsized":"2024-09-10T20:30:10.133864+0000","mapping_epoch":33,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4553296,"num_objects":556,"num_object_clones":0,"num_object_copies":2224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11551","reported_seq":15531,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:11.579853+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:11.579853+0000","last_peered":"2024-09-10T20:30:11.579853+0000","last_clean":"2024-09-10T20:30:11.579853+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:11.579853+0000","last_undegraded":"2024-09-10T20:30:11.579853+0000","last_fullsized":"2024-09-10T20:30:11.579853+0000","mapping_epoch":39,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4841472,"num_objects":591,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11482","reported_seq":11764,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:10.118722+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:10.118722+0000","last_peered":"2024-09-10T20:30:10.118722+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:10.118722+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5164784,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11516","reported_seq":11823,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:09.886053+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:09.886053+0000","last_peered":"2024-09-10T20:30:09.886053+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:09.886053+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5315152,"num_objects":649,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11515,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1211}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11270","reported_seq":11463,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.733960+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:11.733960+0000","last_peered":"2024-09-10T20:30:11.733960+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:11.733960+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4594256,"num_objects":561,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1342}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11465","reported_seq":15525,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.861648+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:08.861648+0000","last_peered":"2024-09-10T20:30:08.861648+0000","last_clean":"2024-09-10T20:30:08.861648+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:08.861648+0000","last_undegraded":"2024-09-10T20:30:08.861648+0000","last_fullsized":"2024-09-10T20:30:08.861648+0000","mapping_epoch":43,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5136384,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11862","reported_seq":12077,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.770966+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:11.770966+0000","last_peered":"2024-09-10T20:30:11.770966+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:11.770966+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4805792,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11493","reported_seq":15598,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:08.825681+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:08.825681+0000","last_peered":"2024-09-10T20:30:08.825681+0000","last_clean":"2024-09-10T20:30:08.825681+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:08.825681+0000","last_undegraded":"2024-09-10T20:30:08.825681+0000","last_fullsized":"2024-09-10T20:30:08.825681+0000","mapping_epoch":41,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5160960,"num_objects":630,"num_object_clones":0,"num_object_copies":2520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2366,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80177864,"num_objects":9791,"num_object_clones":0,"num_object_copies":39164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144379,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160612352,"data_stored":160612352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711800,"kb_used_data":40092,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93660040,"statfs":{"total":96636764160,"available":95907880960,"internally_reserved":0,"allocated":41054208,"data_stored":40822163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,6,0,4,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701452,"kb_used_data":38588,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93670388,"statfs":{"total":96636764160,"available":95918477312,"internally_reserved":0,"allocated":39514112,"data_stored":39282067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578076,"kb_used_data":41036,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93793764,"statfs":{"total":96636764160,"available":96044814336,"internally_reserved":0,"allocated":42020864,"data_stored":41788819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,2,8,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215238,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699796,"kb_used_data":40644,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93672044,"statfs":{"total":96636764160,"available":95920173056,"internally_reserved":0,"allocated":41619456,"data_stored":41387411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,7,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41230336,"data_stored":41230336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40951808,"data_stored":40951808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38465536,"data_stored":38465536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39964672,"data_stored":39964672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:14.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:14.659 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:14.671 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-10T20:30:14.122240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76696264,"num_objects":9366,"num_object_clones":0,"num_object_copies":37464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144804,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691932,"kb_used_data":152836,"kb_used_omap":14,"kb_used_meta":2538993,"kb_avail":374795428,"statfs":{"total":386547056640,"available":383790518272,"internally_reserved":0,"allocated":156504064,"data_stored":155575884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2599929398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,10,7,9,12,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20299776,"num_objects":-2478,"num_object_clones":0,"num_object_copies":-9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008463"},"pg_stats":[{"pgid":"1.f","version":"44'11520","reported_seq":11811,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:09.795086+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:09.795086+0000","last_peered":"2024-09-10T20:30:09.795086+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:09.795086+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4931584,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11719","reported_seq":11860,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.958874+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:13.958874+0000","last_peered":"2024-09-10T20:30:13.958874+0000","last_clean":"2024-09-10T20:30:13.958874+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:13.958874+0000","last_undegraded":"2024-09-10T20:30:13.958874+0000","last_fullsized":"2024-09-10T20:30:13.958874+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4366336,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11570","reported_seq":11781,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.642821+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:11.642821+0000","last_peered":"2024-09-10T20:30:11.642821+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:11.642821+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4668008,"num_objects":571,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11569,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11825","reported_seq":16004,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.929936+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:13.929936+0000","last_peered":"2024-09-10T20:30:13.929936+0000","last_clean":"2024-09-10T20:30:13.929936+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:13.929936+0000","last_undegraded":"2024-09-10T20:30:13.929936+0000","last_fullsized":"2024-09-10T20:30:13.929936+0000","mapping_epoch":37,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4759552,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11506","reported_seq":11836,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:10.026231+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:10.026231+0000","last_peered":"2024-09-10T20:30:10.026231+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:10.026231+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428384,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11477","reported_seq":11708,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:10.132824+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:10.132824+0000","last_peered":"2024-09-10T20:30:10.132824+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:10.132824+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5087232,"num_objects":621,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11542","reported_seq":12839,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:13.599253+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:13.599253+0000","last_peered":"2024-09-10T20:30:13.599253+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:13.599253+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4322464,"num_objects":528,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11708","reported_seq":11886,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:13.591111+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:13.591111+0000","last_peered":"2024-09-10T20:30:13.591111+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:13.591111+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4789408,"num_objects":585,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11445","reported_seq":15427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:10.133864+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:10.133864+0000","last_peered":"2024-09-10T20:30:10.133864+0000","last_clean":"2024-09-10T20:30:10.133864+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:10.133864+0000","last_undegraded":"2024-09-10T20:30:10.133864+0000","last_fullsized":"2024-09-10T20:30:10.133864+0000","mapping_epoch":33,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4553296,"num_objects":556,"num_object_clones":0,"num_object_copies":2224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11551","reported_seq":15531,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:11.579853+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:11.579853+0000","last_peered":"2024-09-10T20:30:11.579853+0000","last_clean":"2024-09-10T20:30:11.579853+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:11.579853+0000","last_undegraded":"2024-09-10T20:30:11.579853+0000","last_fullsized":"2024-09-10T20:30:11.579853+0000","mapping_epoch":39,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4841472,"num_objects":591,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11482","reported_seq":11764,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:10.118722+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:10.118722+0000","last_peered":"2024-09-10T20:30:10.118722+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:10.118722+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5164784,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11516","reported_seq":11823,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:09.886053+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:09.886053+0000","last_peered":"2024-09-10T20:30:09.886053+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:09.886053+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5315152,"num_objects":649,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11515,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1211}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11270","reported_seq":11463,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.733960+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:11.733960+0000","last_peered":"2024-09-10T20:30:11.733960+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:11.733960+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4594256,"num_objects":561,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1342}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11539","reported_seq":15599,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.947784+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:13.947784+0000","last_peered":"2024-09-10T20:30:13.947784+0000","last_clean":"2024-09-10T20:30:13.947784+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:13.947784+0000","last_undegraded":"2024-09-10T20:30:13.947784+0000","last_fullsized":"2024-09-10T20:30:13.947784+0000","mapping_epoch":43,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530176,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11862","reported_seq":12077,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.770966+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:11.770966+0000","last_peered":"2024-09-10T20:30:11.770966+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:11.770966+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4805792,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11569","reported_seq":15674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:14.028094+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:14.028094+0000","last_peered":"2024-09-10T20:30:14.028094+0000","last_clean":"2024-09-10T20:30:14.028094+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:14.028094+0000","last_undegraded":"2024-09-10T20:30:14.028094+0000","last_fullsized":"2024-09-10T20:30:14.028094+0000","mapping_epoch":41,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4538368,"num_objects":554,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":76696264,"num_objects":9366,"num_object_clones":0,"num_object_copies":37464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144804,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":152186880,"data_stored":152186880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711800,"kb_used_data":40092,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93660040,"statfs":{"total":96636764160,"available":95907880960,"internally_reserved":0,"allocated":41054208,"data_stored":40822163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,6,0,4,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701452,"kb_used_data":38588,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93670388,"statfs":{"total":96636764160,"available":95918477312,"internally_reserved":0,"allocated":39514112,"data_stored":39282067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578124,"kb_used_data":36924,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93793716,"statfs":{"total":96636764160,"available":96044765184,"internally_reserved":0,"allocated":37810176,"data_stored":37578131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554168018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700556,"kb_used_data":37232,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93671284,"statfs":{"total":96636764160,"available":95919394816,"internally_reserved":0,"allocated":38125568,"data_stored":37893523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679210703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37011456,"data_stored":37011456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36745216,"data_stored":36745216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38465536,"data_stored":38465536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39964672,"data_stored":39964672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:14.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:14.672 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11520", 'reported_seq': 11811, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:09.795086+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:09.795086+0000', 'last_peered': '2024-09-10T20:30:09.795086+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:09.795086+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4931584, 'num_objects': 602, 'num_object_clones': 0, 'num_object_copies': 2408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11520, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1253}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11570", 'reported_seq': 11781, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:11.642821+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:11.642821+0000', 'last_peered': '2024-09-10T20:30:11.642821+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:11.642821+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4668008, 'num_objects': 571, 'num_object_clones': 0, 'num_object_copies': 2284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 571, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11569, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1337}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11506", 'reported_seq': 11836, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:10.026231+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:10.026231+0000', 'last_peered': '2024-09-10T20:30:10.026231+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:10.026231+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5428384, 'num_objects': 663, 'num_object_clones': 0, 'num_object_copies': 2652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11504, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11477", 'reported_seq': 11708, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:10.132824+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:10.132824+0000', 'last_peered': '2024-09-10T20:30:10.132824+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:10.132824+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5087232, 'num_objects': 621, 'num_object_clones': 0, 'num_object_copies': 2484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11477, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1236}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11542", 'reported_seq': 12839, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:13.599253+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:13.599253+0000', 'last_peered': '2024-09-10T20:30:13.599253+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:13.599253+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4322464, 'num_objects': 528, 'num_object_clones': 0, 'num_object_copies': 2112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1353, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11540, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 604, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -825}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11708", 'reported_seq': 11886, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:13.591111+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:13.591111+0000', 'last_peered': '2024-09-10T20:30:13.591111+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:13.591111+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4789408, 'num_objects': 585, 'num_object_clones': 0, 'num_object_copies': 2340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11706, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:14.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11482", 'reported_seq': 11764, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:10.118722+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:10.118722+0000', 'last_peered': '2024-09-10T20:30:10.118722+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:10.118722+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5164784, 'num_objects': 631, 'num_object_clones': 0, 'num_object_copies': 2524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11479, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1225}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11516", 'reported_seq': 11823, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:09.886053+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:09.886053+0000', 'last_peered': '2024-09-10T20:30:09.886053+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:09.886053+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5315152, 'num_objects': 649, 'num_object_clones': 0, 'num_object_copies': 2596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11515, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11270", 'reported_seq': 11463, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:11.733960+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:11.733960+0000', 'last_peered': '2024-09-10T20:30:11.733960+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:11.733960+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4594256, 'num_objects': 561, 'num_object_clones': 0, 'num_object_copies': 2244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11269, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1342}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:14.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11862", 'reported_seq': 12077, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:11.770966+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:11.770966+0000', 'last_peered': '2024-09-10T20:30:11.770966+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:11.770966+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4805792, 'num_objects': 587, 'num_object_clones': 0, 'num_object_copies': 2348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11860, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1321}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:14.674 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:15.450 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:15.450 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18367/37464 objects degraded (49.026%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":704,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9366,"data_bytes":76696264,"bytes_used":2756538368,"bytes_avail":383790518272,"bytes_total":386547056640,"degraded_objects":18367,"degraded_total":37464,"degraded_ratio":0.49025731368780695,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:15.480 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:15.481 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:16.164 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:16.165 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:16.183 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-10T20:30:16.122848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":73714376,"num_objects":9002,"num_object_clones":0,"num_object_copies":36008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145168,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674692,"kb_used_data":148084,"kb_used_omap":14,"kb_used_meta":2526513,"kb_avail":374812668,"statfs":{"total":386547056640,"available":383808172032,"internally_reserved":0,"allocated":151638016,"data_stored":150709836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2587149878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,6,13,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20127744,"num_objects":-2457,"num_object_clones":0,"num_object_copies":-9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2457,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008460"},"pg_stats":[{"pgid":"1.f","version":"44'11597","reported_seq":11887,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.228214+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:15.228214+0000","last_peered":"2024-09-10T20:30:15.228214+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:15.228214+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4300800,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11597,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11719","reported_seq":11860,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.958874+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:13.958874+0000","last_peered":"2024-09-10T20:30:13.958874+0000","last_clean":"2024-09-10T20:30:13.958874+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:13.958874+0000","last_undegraded":"2024-09-10T20:30:13.958874+0000","last_fullsized":"2024-09-10T20:30:13.958874+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4366336,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11570","reported_seq":11781,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.642821+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:11.642821+0000","last_peered":"2024-09-10T20:30:11.642821+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:11.642821+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4668008,"num_objects":571,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11569,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11825","reported_seq":16004,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.929936+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:13.929936+0000","last_peered":"2024-09-10T20:30:13.929936+0000","last_clean":"2024-09-10T20:30:13.929936+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:13.929936+0000","last_undegraded":"2024-09-10T20:30:13.929936+0000","last_fullsized":"2024-09-10T20:30:13.929936+0000","mapping_epoch":37,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4759552,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11567","reported_seq":11897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.123748+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:15.123748+0000","last_peered":"2024-09-10T20:30:15.123748+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:15.123748+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4928672,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11526","reported_seq":11757,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:14.984656+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:14.984656+0000","last_peered":"2024-09-10T20:30:14.984656+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:14.984656+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4685824,"num_objects":572,"num_object_clones":0,"num_object_copies":2288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11526,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11542","reported_seq":12839,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:13.599253+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:13.599253+0000","last_peered":"2024-09-10T20:30:13.599253+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:13.599253+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4322464,"num_objects":528,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11708","reported_seq":11886,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:13.591111+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:13.591111+0000","last_peered":"2024-09-10T20:30:13.591111+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:13.591111+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4789408,"num_objects":585,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11503","reported_seq":15485,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:15.000116+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:15.000116+0000","last_peered":"2024-09-10T20:30:15.000116+0000","last_clean":"2024-09-10T20:30:15.000116+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:15.000116+0000","last_undegraded":"2024-09-10T20:30:15.000116+0000","last_fullsized":"2024-09-10T20:30:15.000116+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4078160,"num_objects":498,"num_object_clones":0,"num_object_copies":1992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11551","reported_seq":15531,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:11.579853+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:11.579853+0000","last_peered":"2024-09-10T20:30:11.579853+0000","last_clean":"2024-09-10T20:30:11.579853+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:11.579853+0000","last_undegraded":"2024-09-10T20:30:11.579853+0000","last_fullsized":"2024-09-10T20:30:11.579853+0000","mapping_epoch":39,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4841472,"num_objects":591,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11534","reported_seq":11817,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.035272+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:15.035272+0000","last_peered":"2024-09-10T20:30:15.035272+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:15.035272+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4738800,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11583","reported_seq":11890,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.200417+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:15.200417+0000","last_peered":"2024-09-10T20:30:15.200417+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:15.200417+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4766288,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11270","reported_seq":11463,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.733960+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:11.733960+0000","last_peered":"2024-09-10T20:30:11.733960+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:11.733960+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4594256,"num_objects":561,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1342}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11539","reported_seq":15599,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.947784+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:13.947784+0000","last_peered":"2024-09-10T20:30:13.947784+0000","last_clean":"2024-09-10T20:30:13.947784+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:13.947784+0000","last_undegraded":"2024-09-10T20:30:13.947784+0000","last_fullsized":"2024-09-10T20:30:13.947784+0000","mapping_epoch":43,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530176,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11862","reported_seq":12077,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:11.770966+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:11.770966+0000","last_peered":"2024-09-10T20:30:11.770966+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:11.770966+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4805792,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11569","reported_seq":15674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:14.028094+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:14.028094+0000","last_peered":"2024-09-10T20:30:14.028094+0000","last_clean":"2024-09-10T20:30:14.028094+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:14.028094+0000","last_undegraded":"2024-09-10T20:30:14.028094+0000","last_fullsized":"2024-09-10T20:30:14.028094+0000","mapping_epoch":41,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4538368,"num_objects":554,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":73714376,"num_objects":9002,"num_object_clones":0,"num_object_copies":36008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145168,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":148021248,"data_stored":148021248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694560,"kb_used_data":35340,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93677280,"statfs":{"total":96636764160,"available":95925534720,"internally_reserved":0,"allocated":36188160,"data_stored":35956115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701452,"kb_used_data":38588,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93670388,"statfs":{"total":96636764160,"available":95918477312,"internally_reserved":0,"allocated":39514112,"data_stored":39282067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578124,"kb_used_data":36924,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93793716,"statfs":{"total":96636764160,"available":96044765184,"internally_reserved":0,"allocated":37810176,"data_stored":37578131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554168018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700556,"kb_used_data":37232,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93671284,"statfs":{"total":96636764160,"available":95919394816,"internally_reserved":0,"allocated":38125568,"data_stored":37893523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679210703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37011456,"data_stored":37011456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36745216,"data_stored":36745216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38465536,"data_stored":38465536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35799040,"data_stored":35799040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:19.185 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:19.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:19.763 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:19.794 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-10T20:30:18.125835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71633608,"num_objects":8748,"num_object_clones":0,"num_object_copies":34992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145422,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674948,"kb_used_data":144176,"kb_used_omap":14,"kb_used_meta":2530673,"kb_avail":374812412,"statfs":{"total":386547056640,"available":383807909888,"internally_reserved":0,"allocated":147636224,"data_stored":146708044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2591409718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,8,10,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19030016,"num_objects":-2323,"num_object_clones":0,"num_object_copies":-9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2323,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010616"},"pg_stats":[{"pgid":"1.f","version":"44'11597","reported_seq":11887,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.228214+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:15.228214+0000","last_peered":"2024-09-10T20:30:15.228214+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:15.228214+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4300800,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11597,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11719","reported_seq":11860,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.958874+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:13.958874+0000","last_peered":"2024-09-10T20:30:13.958874+0000","last_clean":"2024-09-10T20:30:13.958874+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:13.958874+0000","last_undegraded":"2024-09-10T20:30:13.958874+0000","last_fullsized":"2024-09-10T20:30:13.958874+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4366336,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11637","reported_seq":11848,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.886683+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:16.886683+0000","last_peered":"2024-09-10T20:30:16.886683+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:16.886683+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4119144,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11636,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11825","reported_seq":16004,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.929936+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:13.929936+0000","last_peered":"2024-09-10T20:30:13.929936+0000","last_clean":"2024-09-10T20:30:13.929936+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:13.929936+0000","last_undegraded":"2024-09-10T20:30:13.929936+0000","last_fullsized":"2024-09-10T20:30:13.929936+0000","mapping_epoch":37,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4759552,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11567","reported_seq":11897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.123748+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:15.123748+0000","last_peered":"2024-09-10T20:30:15.123748+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:15.123748+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4928672,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11526","reported_seq":11757,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:14.984656+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:14.984656+0000","last_peered":"2024-09-10T20:30:14.984656+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:14.984656+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4685824,"num_objects":572,"num_object_clones":0,"num_object_copies":2288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11526,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11542","reported_seq":12839,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:13.599253+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:13.599253+0000","last_peered":"2024-09-10T20:30:13.599253+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:13.599253+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4322464,"num_objects":528,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11708","reported_seq":11886,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:13.591111+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:13.591111+0000","last_peered":"2024-09-10T20:30:13.591111+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:13.591111+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4789408,"num_objects":585,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11503","reported_seq":15485,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:15.000116+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:15.000116+0000","last_peered":"2024-09-10T20:30:15.000116+0000","last_clean":"2024-09-10T20:30:15.000116+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:15.000116+0000","last_undegraded":"2024-09-10T20:30:15.000116+0000","last_fullsized":"2024-09-10T20:30:15.000116+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4078160,"num_objects":498,"num_object_clones":0,"num_object_copies":1992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11609","reported_seq":15589,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:16.771303+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:16.771303+0000","last_peered":"2024-09-10T20:30:16.771303+0000","last_clean":"2024-09-10T20:30:16.771303+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:16.771303+0000","last_undegraded":"2024-09-10T20:30:16.771303+0000","last_fullsized":"2024-09-10T20:30:16.771303+0000","mapping_epoch":39,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4366336,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11534","reported_seq":11817,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.035272+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:15.035272+0000","last_peered":"2024-09-10T20:30:15.035272+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:15.035272+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4738800,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11583","reported_seq":11890,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.200417+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:15.200417+0000","last_peered":"2024-09-10T20:30:15.200417+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:15.200417+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4766288,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11336","reported_seq":11529,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.820226+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:16.820226+0000","last_peered":"2024-09-10T20:30:16.820226+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:16.820226+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4053584,"num_objects":495,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11539","reported_seq":15599,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:13.947784+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:13.947784+0000","last_peered":"2024-09-10T20:30:13.947784+0000","last_clean":"2024-09-10T20:30:13.947784+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:13.947784+0000","last_undegraded":"2024-09-10T20:30:13.947784+0000","last_fullsized":"2024-09-10T20:30:13.947784+0000","mapping_epoch":43,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530176,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11925","reported_seq":12140,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.759460+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:16.759460+0000","last_peered":"2024-09-10T20:30:16.759460+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:16.759460+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4289696,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11569","reported_seq":15674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:14.028094+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:14.028094+0000","last_peered":"2024-09-10T20:30:14.028094+0000","last_clean":"2024-09-10T20:30:14.028094+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:14.028094+0000","last_undegraded":"2024-09-10T20:30:14.028094+0000","last_fullsized":"2024-09-10T20:30:14.028094+0000","mapping_epoch":41,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4538368,"num_objects":554,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":71633608,"num_objects":8748,"num_object_clones":0,"num_object_copies":34992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145422,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144007168,"data_stored":144007168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694560,"kb_used_data":35340,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93677280,"statfs":{"total":96636764160,"available":95925534720,"internally_reserved":0,"allocated":36188160,"data_stored":35956115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701708,"kb_used_data":34680,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93670132,"statfs":{"total":96636764160,"available":95918215168,"internally_reserved":0,"allocated":35512320,"data_stored":35280275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578124,"kb_used_data":36924,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93793716,"statfs":{"total":96636764160,"available":96044765184,"internally_reserved":0,"allocated":37810176,"data_stored":37578131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554168018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700556,"kb_used_data":37232,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93671284,"statfs":{"total":96636764160,"available":95919394816,"internally_reserved":0,"allocated":38125568,"data_stored":37893523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679210703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37011456,"data_stored":37011456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36745216,"data_stored":36745216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34451456,"data_stored":34451456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35799040,"data_stored":35799040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:19.795 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:20.379 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:20.380 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:20.398 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-10T20:30:20.126275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":68750024,"num_objects":8396,"num_object_clones":0,"num_object_copies":33584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145774,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674508,"kb_used_data":135424,"kb_used_omap":14,"kb_used_meta":2538993,"kb_avail":374812852,"statfs":{"total":386547056640,"available":383808360448,"internally_reserved":0,"allocated":138674176,"data_stored":137745996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2599929398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,6,12,11,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19759104,"num_objects":-2412,"num_object_clones":0,"num_object_copies":-9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006535"},"pg_stats":[{"pgid":"1.f","version":"44'11597","reported_seq":11887,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.228214+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:15.228214+0000","last_peered":"2024-09-10T20:30:15.228214+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:15.228214+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4300800,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11597,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11780","reported_seq":11921,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.613146+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:18.613146+0000","last_peered":"2024-09-10T20:30:18.613146+0000","last_clean":"2024-09-10T20:30:18.613146+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:18.613146+0000","last_undegraded":"2024-09-10T20:30:18.613146+0000","last_fullsized":"2024-09-10T20:30:18.613146+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11637","reported_seq":11848,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.886683+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:16.886683+0000","last_peered":"2024-09-10T20:30:16.886683+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:16.886683+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4119144,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11636,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11887","reported_seq":16065,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.621206+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:18.621206+0000","last_peered":"2024-09-10T20:30:18.621206+0000","last_clean":"2024-09-10T20:30:18.621206+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:18.621206+0000","last_undegraded":"2024-09-10T20:30:18.621206+0000","last_fullsized":"2024-09-10T20:30:18.621206+0000","mapping_epoch":37,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4251648,"num_objects":519,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11567","reported_seq":11897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.123748+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:15.123748+0000","last_peered":"2024-09-10T20:30:15.123748+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:15.123748+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4928672,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11526","reported_seq":11757,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:14.984656+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:14.984656+0000","last_peered":"2024-09-10T20:30:14.984656+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:14.984656+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4685824,"num_objects":572,"num_object_clones":0,"num_object_copies":2288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11526,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11602","reported_seq":13042,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:18.603204+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:18.603204+0000","last_peered":"2024-09-10T20:30:18.603204+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:18.603204+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3830944,"num_objects":468,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11771","reported_seq":11948,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:18.544003+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:18.544003+0000","last_peered":"2024-09-10T20:30:18.544003+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:18.544003+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4273312,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11503","reported_seq":15485,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:15.000116+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:15.000116+0000","last_peered":"2024-09-10T20:30:15.000116+0000","last_clean":"2024-09-10T20:30:15.000116+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:15.000116+0000","last_undegraded":"2024-09-10T20:30:15.000116+0000","last_fullsized":"2024-09-10T20:30:15.000116+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4078160,"num_objects":498,"num_object_clones":0,"num_object_copies":1992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11609","reported_seq":15589,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:16.771303+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:16.771303+0000","last_peered":"2024-09-10T20:30:16.771303+0000","last_clean":"2024-09-10T20:30:16.771303+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:16.771303+0000","last_undegraded":"2024-09-10T20:30:16.771303+0000","last_fullsized":"2024-09-10T20:30:16.771303+0000","mapping_epoch":39,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4366336,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11534","reported_seq":11817,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.035272+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:15.035272+0000","last_peered":"2024-09-10T20:30:15.035272+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:15.035272+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4738800,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11583","reported_seq":11890,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.200417+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:15.200417+0000","last_peered":"2024-09-10T20:30:15.200417+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:15.200417+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4766288,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11336","reported_seq":11529,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.820226+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:16.820226+0000","last_peered":"2024-09-10T20:30:16.820226+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:16.820226+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4053584,"num_objects":495,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11592","reported_seq":15652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.586338+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:18.586338+0000","last_peered":"2024-09-10T20:30:18.586338+0000","last_clean":"2024-09-10T20:30:18.586338+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:18.586338+0000","last_undegraded":"2024-09-10T20:30:18.586338+0000","last_fullsized":"2024-09-10T20:30:18.586338+0000","mapping_epoch":43,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096000,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":544,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11925","reported_seq":12140,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.759460+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:16.759460+0000","last_peered":"2024-09-10T20:30:16.759460+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:16.759460+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4289696,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11622","reported_seq":15727,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.552876+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:18.552876+0000","last_peered":"2024-09-10T20:30:18.552876+0000","last_clean":"2024-09-10T20:30:18.552876+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:18.552876+0000","last_undegraded":"2024-09-10T20:30:18.552876+0000","last_fullsized":"2024-09-10T20:30:18.552876+0000","mapping_epoch":41,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4104192,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":68750024,"num_objects":8396,"num_object_clones":0,"num_object_copies":33584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145774,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136036352,"data_stored":136036352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694560,"kb_used_data":35340,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93677280,"statfs":{"total":96636764160,"available":95925534720,"internally_reserved":0,"allocated":36188160,"data_stored":35956115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701708,"kb_used_data":34680,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93670132,"statfs":{"total":96636764160,"available":95918215168,"internally_reserved":0,"allocated":35512320,"data_stored":35280275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578280,"kb_used_data":32916,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93793560,"statfs":{"total":96636764160,"available":96044605440,"internally_reserved":0,"allocated":33705984,"data_stored":33473939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558427858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699960,"kb_used_data":32488,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93671880,"statfs":{"total":96636764160,"available":95920005120,"internally_reserved":0,"allocated":33267712,"data_stored":33035667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":683470543},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32886784,"data_stored":32886784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32899072,"data_stored":32899072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34451456,"data_stored":34451456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35799040,"data_stored":35799040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:20.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:20.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11597", 'reported_seq': 11887, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:15.228214+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:15.228214+0000', 'last_peered': '2024-09-10T20:30:15.228214+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:15.228214+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4300800, 'num_objects': 525, 'num_object_clones': 0, 'num_object_copies': 2100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11597, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1330}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:20.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11637", 'reported_seq': 11848, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:16.886683+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:16.886683+0000', 'last_peered': '2024-09-10T20:30:16.886683+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:16.886683+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4119144, 'num_objects': 504, 'num_object_clones': 0, 'num_object_copies': 2016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 504, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11636, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1404}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11567", 'reported_seq': 11897, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:15.123748+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:15.123748+0000', 'last_peered': '2024-09-10T20:30:15.123748+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:15.123748+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4928672, 'num_objects': 602, 'num_object_clones': 0, 'num_object_copies': 2408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11565, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1253}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11526", 'reported_seq': 11757, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:14.984656+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:14.984656+0000', 'last_peered': '2024-09-10T20:30:14.984656+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:14.984656+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4685824, 'num_objects': 572, 'num_object_clones': 0, 'num_object_copies': 2288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11526, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1285}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11602", 'reported_seq': 13042, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:18.603204+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:18.603204+0000', 'last_peered': '2024-09-10T20:30:18.603204+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:18.603204+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3830944, 'num_objects': 468, 'num_object_clones': 0, 'num_object_copies': 1872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1282, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 675, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -814}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11771", 'reported_seq': 11948, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:18.544003+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:18.544003+0000', 'last_peered': '2024-09-10T20:30:18.544003+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:18.544003+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4273312, 'num_objects': 522, 'num_object_clones': 0, 'num_object_copies': 2088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11769, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:20.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:20.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11534", 'reported_seq': 11817, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:15.035272+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:15.035272+0000', 'last_peered': '2024-09-10T20:30:15.035272+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:15.035272+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4738800, 'num_objects': 579, 'num_object_clones': 0, 'num_object_copies': 2316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11531, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1277}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:20.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:20.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11583", 'reported_seq': 11890, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:15.200417+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:15.200417+0000', 'last_peered': '2024-09-10T20:30:15.200417+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:15.200417+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4766288, 'num_objects': 582, 'num_object_clones': 0, 'num_object_copies': 2328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11582, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1278}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:20.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:20.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11336", 'reported_seq': 11529, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:16.820226+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:16.820226+0000', 'last_peered': '2024-09-10T20:30:16.820226+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:16.820226+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4053584, 'num_objects': 495, 'num_object_clones': 0, 'num_object_copies': 1980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11335, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:20.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:20.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11925", 'reported_seq': 12140, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:16.759460+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:16.759460+0000', 'last_peered': '2024-09-10T20:30:16.759460+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:16.759460+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4289696, 'num_objects': 524, 'num_object_clones': 0, 'num_object_copies': 2096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11923, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:20.401 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:21.153 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:21.154 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18296/33584 objects degraded (54.478%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":709,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8396,"data_bytes":68750024,"bytes_used":2738696192,"bytes_avail":383808360448,"bytes_total":386547056640,"degraded_objects":18296,"degraded_total":33584,"degraded_ratio":0.54478323010957597,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:21.177 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:21.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:21.772 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:21.772 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:21.796 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-10T20:30:20.126275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":68750024,"num_objects":8396,"num_object_clones":0,"num_object_copies":33584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145774,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674508,"kb_used_data":135424,"kb_used_omap":14,"kb_used_meta":2538993,"kb_avail":374812852,"statfs":{"total":386547056640,"available":383808360448,"internally_reserved":0,"allocated":138674176,"data_stored":137745996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2599929398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,6,12,11,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19759104,"num_objects":-2412,"num_object_clones":0,"num_object_copies":-9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006535"},"pg_stats":[{"pgid":"1.f","version":"44'11597","reported_seq":11887,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.228214+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:15.228214+0000","last_peered":"2024-09-10T20:30:15.228214+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:15.228214+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4300800,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11597,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11780","reported_seq":11921,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.613146+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:18.613146+0000","last_peered":"2024-09-10T20:30:18.613146+0000","last_clean":"2024-09-10T20:30:18.613146+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:18.613146+0000","last_undegraded":"2024-09-10T20:30:18.613146+0000","last_fullsized":"2024-09-10T20:30:18.613146+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11637","reported_seq":11848,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.886683+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:16.886683+0000","last_peered":"2024-09-10T20:30:16.886683+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:16.886683+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4119144,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11636,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11887","reported_seq":16065,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.621206+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:18.621206+0000","last_peered":"2024-09-10T20:30:18.621206+0000","last_clean":"2024-09-10T20:30:18.621206+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:18.621206+0000","last_undegraded":"2024-09-10T20:30:18.621206+0000","last_fullsized":"2024-09-10T20:30:18.621206+0000","mapping_epoch":37,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4251648,"num_objects":519,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11567","reported_seq":11897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.123748+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:15.123748+0000","last_peered":"2024-09-10T20:30:15.123748+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:15.123748+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4928672,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11526","reported_seq":11757,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:14.984656+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:14.984656+0000","last_peered":"2024-09-10T20:30:14.984656+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:14.984656+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4685824,"num_objects":572,"num_object_clones":0,"num_object_copies":2288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11526,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11602","reported_seq":13042,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:18.603204+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:18.603204+0000","last_peered":"2024-09-10T20:30:18.603204+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:18.603204+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3830944,"num_objects":468,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11771","reported_seq":11948,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:18.544003+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:18.544003+0000","last_peered":"2024-09-10T20:30:18.544003+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:18.544003+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4273312,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11503","reported_seq":15485,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:15.000116+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:15.000116+0000","last_peered":"2024-09-10T20:30:15.000116+0000","last_clean":"2024-09-10T20:30:15.000116+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:15.000116+0000","last_undegraded":"2024-09-10T20:30:15.000116+0000","last_fullsized":"2024-09-10T20:30:15.000116+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4078160,"num_objects":498,"num_object_clones":0,"num_object_copies":1992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11609","reported_seq":15589,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:16.771303+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:16.771303+0000","last_peered":"2024-09-10T20:30:16.771303+0000","last_clean":"2024-09-10T20:30:16.771303+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:16.771303+0000","last_undegraded":"2024-09-10T20:30:16.771303+0000","last_fullsized":"2024-09-10T20:30:16.771303+0000","mapping_epoch":39,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4366336,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11534","reported_seq":11817,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.035272+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:15.035272+0000","last_peered":"2024-09-10T20:30:15.035272+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:15.035272+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4738800,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11583","reported_seq":11890,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:15.200417+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:15.200417+0000","last_peered":"2024-09-10T20:30:15.200417+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:15.200417+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4766288,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11336","reported_seq":11529,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.820226+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:16.820226+0000","last_peered":"2024-09-10T20:30:16.820226+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:16.820226+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4053584,"num_objects":495,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11592","reported_seq":15652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.586338+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:18.586338+0000","last_peered":"2024-09-10T20:30:18.586338+0000","last_clean":"2024-09-10T20:30:18.586338+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:18.586338+0000","last_undegraded":"2024-09-10T20:30:18.586338+0000","last_fullsized":"2024-09-10T20:30:18.586338+0000","mapping_epoch":43,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096000,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":544,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11925","reported_seq":12140,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:16.759460+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:16.759460+0000","last_peered":"2024-09-10T20:30:16.759460+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:16.759460+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4289696,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11622","reported_seq":15727,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.552876+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:18.552876+0000","last_peered":"2024-09-10T20:30:18.552876+0000","last_clean":"2024-09-10T20:30:18.552876+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:18.552876+0000","last_undegraded":"2024-09-10T20:30:18.552876+0000","last_fullsized":"2024-09-10T20:30:18.552876+0000","mapping_epoch":41,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4104192,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":68750024,"num_objects":8396,"num_object_clones":0,"num_object_copies":33584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145774,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136036352,"data_stored":136036352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694560,"kb_used_data":35340,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93677280,"statfs":{"total":96636764160,"available":95925534720,"internally_reserved":0,"allocated":36188160,"data_stored":35956115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701708,"kb_used_data":34680,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93670132,"statfs":{"total":96636764160,"available":95918215168,"internally_reserved":0,"allocated":35512320,"data_stored":35280275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578280,"kb_used_data":32916,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93793560,"statfs":{"total":96636764160,"available":96044605440,"internally_reserved":0,"allocated":33705984,"data_stored":33473939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558427858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699960,"kb_used_data":32488,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93671880,"statfs":{"total":96636764160,"available":95920005120,"internally_reserved":0,"allocated":33267712,"data_stored":33035667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":683470543},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32886784,"data_stored":32886784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32899072,"data_stored":32899072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34451456,"data_stored":34451456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35799040,"data_stored":35799040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:24.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:25.411 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:25.449 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-10T20:30:24.127804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62950088,"num_objects":7688,"num_object_clones":0,"num_object_copies":30752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146482,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161079,"ondisk_log_size":161079,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2675928,"kb_used_data":124356,"kb_used_omap":14,"kb_used_meta":2551473,"kb_avail":374811432,"statfs":{"total":386547056640,"available":383806906368,"internally_reserved":0,"allocated":127340544,"data_stored":126412364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2612708918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,3,14,11,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17227776,"num_objects":-2103,"num_object_clones":0,"num_object_copies":-8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006369"},"pg_stats":[{"pgid":"1.f","version":"44'11658","reported_seq":11949,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.278875+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:20.278875+0000","last_peered":"2024-09-10T20:30:20.278875+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:20.278875+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":464,"num_object_clones":0,"num_object_copies":1856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11658,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11780","reported_seq":11921,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.613146+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:18.613146+0000","last_peered":"2024-09-10T20:30:18.613146+0000","last_clean":"2024-09-10T20:30:18.613146+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:18.613146+0000","last_undegraded":"2024-09-10T20:30:18.613146+0000","last_fullsized":"2024-09-10T20:30:18.613146+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11684","reported_seq":11897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.879918+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:21.879918+0000","last_peered":"2024-09-10T20:30:21.879918+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:21.879918+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3734120,"num_objects":457,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11683,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11887","reported_seq":16065,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.621206+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:18.621206+0000","last_peered":"2024-09-10T20:30:18.621206+0000","last_clean":"2024-09-10T20:30:18.621206+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:18.621206+0000","last_undegraded":"2024-09-10T20:30:18.621206+0000","last_fullsized":"2024-09-10T20:30:18.621206+0000","mapping_epoch":37,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4251648,"num_objects":519,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11630","reported_seq":11960,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.327707+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:20.327707+0000","last_peered":"2024-09-10T20:30:20.327707+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:20.327707+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4412576,"num_objects":539,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11628,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11590","reported_seq":11821,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.310294+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:20.310294+0000","last_peered":"2024-09-10T20:30:20.310294+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:20.310294+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4161536,"num_objects":508,"num_object_clones":0,"num_object_copies":2032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11651","reported_seq":13367,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:23.722721+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:23.722721+0000","last_peered":"2024-09-10T20:30:23.722721+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:23.722721+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3429536,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":813,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11827","reported_seq":12002,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:23.729797+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:23.729797+0000","last_peered":"2024-09-10T20:30:23.729797+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:23.729797+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814560,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11825,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11559","reported_seq":15541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:20.129463+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:20.129463+0000","last_peered":"2024-09-10T20:30:20.129463+0000","last_clean":"2024-09-10T20:30:20.129463+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:20.129463+0000","last_undegraded":"2024-09-10T20:30:20.129463+0000","last_fullsized":"2024-09-10T20:30:20.129463+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3619408,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11676","reported_seq":15655,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:21.860294+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:21.860294+0000","last_peered":"2024-09-10T20:30:21.860294+0000","last_clean":"2024-09-10T20:30:21.860294+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:21.860294+0000","last_undegraded":"2024-09-10T20:30:21.860294+0000","last_fullsized":"2024-09-10T20:30:21.860294+0000","mapping_epoch":39,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3817472,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4314,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11595","reported_seq":11877,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.188344+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:20.188344+0000","last_peered":"2024-09-10T20:30:20.188344+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:20.188344+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4239088,"num_objects":518,"num_object_clones":0,"num_object_copies":2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11642","reported_seq":11948,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.277580+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:20.277580+0000","last_peered":"2024-09-10T20:30:20.277580+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:20.277580+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4282960,"num_objects":523,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11641,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11399","reported_seq":11592,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.886357+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:21.886357+0000","last_peered":"2024-09-10T20:30:21.886357+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:21.886357+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3537488,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11398,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11592","reported_seq":15652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.586338+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:18.586338+0000","last_peered":"2024-09-10T20:30:18.586338+0000","last_clean":"2024-09-10T20:30:18.586338+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:18.586338+0000","last_undegraded":"2024-09-10T20:30:18.586338+0000","last_fullsized":"2024-09-10T20:30:18.586338+0000","mapping_epoch":43,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096000,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":544,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11987","reported_seq":12202,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.884020+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:21.884020+0000","last_peered":"2024-09-10T20:30:21.884020+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:21.884020+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781792,"num_objects":462,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11622","reported_seq":15727,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.552876+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:18.552876+0000","last_peered":"2024-09-10T20:30:18.552876+0000","last_clean":"2024-09-10T20:30:18.552876+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:18.552876+0000","last_undegraded":"2024-09-10T20:30:18.552876+0000","last_fullsized":"2024-09-10T20:30:18.552876+0000","mapping_epoch":41,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4104192,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":62950088,"num_objects":7688,"num_object_clones":0,"num_object_copies":30752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146482,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124514304,"data_stored":124514304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161079,"ondisk_log_size":161079,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695004,"kb_used_data":31616,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93676836,"statfs":{"total":96636764160,"available":95925080064,"internally_reserved":0,"allocated":32374784,"data_stored":32142739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,11,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702200,"kb_used_data":31012,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93669640,"statfs":{"total":96636764160,"available":95917711360,"internally_reserved":0,"allocated":31756288,"data_stored":31524243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578280,"kb_used_data":32916,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93793560,"statfs":{"total":96636764160,"available":96044605440,"internally_reserved":0,"allocated":33705984,"data_stored":33473939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558427858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700444,"kb_used_data":28812,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93671396,"statfs":{"total":96636764160,"available":95919509504,"internally_reserved":0,"allocated":29503488,"data_stored":29271443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29048832,"data_stored":29048832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32899072,"data_stored":32899072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30658560,"data_stored":30658560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31907840,"data_stored":31907840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:25.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:26.071 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:26.072 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:26.097 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-10T20:30:24.127804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62950088,"num_objects":7688,"num_object_clones":0,"num_object_copies":30752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146482,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161079,"ondisk_log_size":161079,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2675928,"kb_used_data":124356,"kb_used_omap":14,"kb_used_meta":2551473,"kb_avail":374811432,"statfs":{"total":386547056640,"available":383806906368,"internally_reserved":0,"allocated":127340544,"data_stored":126412364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2612708918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,3,14,11,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17227776,"num_objects":-2103,"num_object_clones":0,"num_object_copies":-8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006369"},"pg_stats":[{"pgid":"1.f","version":"44'11658","reported_seq":11949,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.278875+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:20.278875+0000","last_peered":"2024-09-10T20:30:20.278875+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:20.278875+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":464,"num_object_clones":0,"num_object_copies":1856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11658,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11780","reported_seq":11921,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.613146+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:18.613146+0000","last_peered":"2024-09-10T20:30:18.613146+0000","last_clean":"2024-09-10T20:30:18.613146+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:18.613146+0000","last_undegraded":"2024-09-10T20:30:18.613146+0000","last_fullsized":"2024-09-10T20:30:18.613146+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11684","reported_seq":11897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.879918+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:21.879918+0000","last_peered":"2024-09-10T20:30:21.879918+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:21.879918+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3734120,"num_objects":457,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11683,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11887","reported_seq":16065,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.621206+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:18.621206+0000","last_peered":"2024-09-10T20:30:18.621206+0000","last_clean":"2024-09-10T20:30:18.621206+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:18.621206+0000","last_undegraded":"2024-09-10T20:30:18.621206+0000","last_fullsized":"2024-09-10T20:30:18.621206+0000","mapping_epoch":37,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4251648,"num_objects":519,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11630","reported_seq":11960,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.327707+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:20.327707+0000","last_peered":"2024-09-10T20:30:20.327707+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:20.327707+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4412576,"num_objects":539,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11628,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11590","reported_seq":11821,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.310294+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:20.310294+0000","last_peered":"2024-09-10T20:30:20.310294+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:20.310294+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4161536,"num_objects":508,"num_object_clones":0,"num_object_copies":2032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11651","reported_seq":13367,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:23.722721+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:23.722721+0000","last_peered":"2024-09-10T20:30:23.722721+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:23.722721+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3429536,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":813,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11827","reported_seq":12002,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:23.729797+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:23.729797+0000","last_peered":"2024-09-10T20:30:23.729797+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:23.729797+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814560,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11825,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11559","reported_seq":15541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:20.129463+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:20.129463+0000","last_peered":"2024-09-10T20:30:20.129463+0000","last_clean":"2024-09-10T20:30:20.129463+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:20.129463+0000","last_undegraded":"2024-09-10T20:30:20.129463+0000","last_fullsized":"2024-09-10T20:30:20.129463+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3619408,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11676","reported_seq":15655,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:21.860294+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:21.860294+0000","last_peered":"2024-09-10T20:30:21.860294+0000","last_clean":"2024-09-10T20:30:21.860294+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:21.860294+0000","last_undegraded":"2024-09-10T20:30:21.860294+0000","last_fullsized":"2024-09-10T20:30:21.860294+0000","mapping_epoch":39,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3817472,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4314,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11595","reported_seq":11877,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.188344+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:20.188344+0000","last_peered":"2024-09-10T20:30:20.188344+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:20.188344+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4239088,"num_objects":518,"num_object_clones":0,"num_object_copies":2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11642","reported_seq":11948,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:20.277580+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:20.277580+0000","last_peered":"2024-09-10T20:30:20.277580+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:20.277580+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4282960,"num_objects":523,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11641,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11399","reported_seq":11592,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.886357+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:21.886357+0000","last_peered":"2024-09-10T20:30:21.886357+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:21.886357+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3537488,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11398,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11592","reported_seq":15652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.586338+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:18.586338+0000","last_peered":"2024-09-10T20:30:18.586338+0000","last_clean":"2024-09-10T20:30:18.586338+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:18.586338+0000","last_undegraded":"2024-09-10T20:30:18.586338+0000","last_fullsized":"2024-09-10T20:30:18.586338+0000","mapping_epoch":43,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096000,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":544,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11987","reported_seq":12202,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.884020+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:21.884020+0000","last_peered":"2024-09-10T20:30:21.884020+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:21.884020+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781792,"num_objects":462,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11622","reported_seq":15727,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:18.552876+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:18.552876+0000","last_peered":"2024-09-10T20:30:18.552876+0000","last_clean":"2024-09-10T20:30:18.552876+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:18.552876+0000","last_undegraded":"2024-09-10T20:30:18.552876+0000","last_fullsized":"2024-09-10T20:30:18.552876+0000","mapping_epoch":41,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4104192,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":62950088,"num_objects":7688,"num_object_clones":0,"num_object_copies":30752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146482,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124514304,"data_stored":124514304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161079,"ondisk_log_size":161079,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695004,"kb_used_data":31616,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93676836,"statfs":{"total":96636764160,"available":95925080064,"internally_reserved":0,"allocated":32374784,"data_stored":32142739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,11,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702200,"kb_used_data":31012,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93669640,"statfs":{"total":96636764160,"available":95917711360,"internally_reserved":0,"allocated":31756288,"data_stored":31524243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578280,"kb_used_data":32916,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93793560,"statfs":{"total":96636764160,"available":96044605440,"internally_reserved":0,"allocated":33705984,"data_stored":33473939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558427858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700444,"kb_used_data":28812,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93671396,"statfs":{"total":96636764160,"available":95919509504,"internally_reserved":0,"allocated":29503488,"data_stored":29271443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29048832,"data_stored":29048832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32899072,"data_stored":32899072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30658560,"data_stored":30658560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31907840,"data_stored":31907840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:26.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:26.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11658", 'reported_seq': 11949, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:20.278875+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:20.278875+0000', 'last_peered': '2024-09-10T20:30:20.278875+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:20.278875+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3801088, 'num_objects': 464, 'num_object_clones': 0, 'num_object_copies': 1856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11658, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1391}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:26.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:26.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11684", 'reported_seq': 11897, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:21.879918+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:21.879918+0000', 'last_peered': '2024-09-10T20:30:21.879918+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:21.879918+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3734120, 'num_objects': 457, 'num_object_clones': 0, 'num_object_copies': 1828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 457, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11683, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1451}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:26.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:26.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11630", 'reported_seq': 11960, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:20.327707+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:20.327707+0000', 'last_peered': '2024-09-10T20:30:20.327707+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:20.327707+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4412576, 'num_objects': 539, 'num_object_clones': 0, 'num_object_copies': 2156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11628, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:26.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:26.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11590", 'reported_seq': 11821, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:20.310294+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:20.310294+0000', 'last_peered': '2024-09-10T20:30:20.310294+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:20.310294+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4161536, 'num_objects': 508, 'num_object_clones': 0, 'num_object_copies': 2032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11590, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1349}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11651", 'reported_seq': 13367, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:23.722721+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:23.722721+0000', 'last_peered': '2024-09-10T20:30:23.722721+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:23.722721+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3429536, 'num_objects': 419, 'num_object_clones': 0, 'num_object_copies': 1676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1144, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11649, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 813, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -725}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11827", 'reported_seq': 12002, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:23.729797+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:23.729797+0000', 'last_peered': '2024-09-10T20:30:23.729797+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:23.729797+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3814560, 'num_objects': 466, 'num_object_clones': 0, 'num_object_copies': 1864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11825, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11595", 'reported_seq': 11877, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:20.188344+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:20.188344+0000', 'last_peered': '2024-09-10T20:30:20.188344+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:20.188344+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4239088, 'num_objects': 518, 'num_object_clones': 0, 'num_object_copies': 2072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11592, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1338}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11642", 'reported_seq': 11948, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:20.277580+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:20.277580+0000', 'last_peered': '2024-09-10T20:30:20.277580+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:20.277580+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4282960, 'num_objects': 523, 'num_object_clones': 0, 'num_object_copies': 2092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11641, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11399", 'reported_seq': 11592, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:21.886357+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:21.886357+0000', 'last_peered': '2024-09-10T20:30:21.886357+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:21.886357+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3537488, 'num_objects': 432, 'num_object_clones': 0, 'num_object_copies': 1728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11398, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:26.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:26.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'11987", 'reported_seq': 12202, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:21.884020+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:21.884020+0000', 'last_peered': '2024-09-10T20:30:21.884020+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:21.884020+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3781792, 'num_objects': 462, 'num_object_clones': 0, 'num_object_copies': 1848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:26.101 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:27.038 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:27.038 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18158/28136 objects degraded (64.537%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":715,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7034,"data_bytes":57592520,"bytes_used":2723565568,"bytes_avail":383823491072,"bytes_total":386547056640,"degraded_objects":18158,"degraded_total":28136,"degraded_ratio":0.64536536821154389,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:27.062 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:27.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:27.666 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:27.667 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:27.695 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-10T20:30:26.128619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":57592520,"num_objects":7034,"num_object_clones":0,"num_object_copies":28136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7034,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147136,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2659732,"kb_used_data":116496,"kb_used_omap":14,"kb_used_meta":2543153,"kb_avail":374827628,"statfs":{"total":386547056640,"available":383823491072,"internally_reserved":0,"allocated":119291904,"data_stored":118363724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2604189238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,9,0,8,16,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19103744,"num_objects":-2332,"num_object_clones":0,"num_object_copies":-9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2332,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006378"},"pg_stats":[{"pgid":"1.f","version":"44'11722","reported_seq":12013,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.261291+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:25.261291+0000","last_peered":"2024-09-10T20:30:25.261291+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:25.261291+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":400,"num_object_clones":0,"num_object_copies":1600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11722,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11847","reported_seq":11988,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:24.100173+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:24.100173+0000","last_peered":"2024-09-10T20:30:24.100173+0000","last_clean":"2024-09-10T20:30:24.100173+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:24.100173+0000","last_undegraded":"2024-09-10T20:30:24.100173+0000","last_fullsized":"2024-09-10T20:30:24.100173+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":405,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8288,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11684","reported_seq":11897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.879918+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:21.879918+0000","last_peered":"2024-09-10T20:30:21.879918+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:21.879918+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3734120,"num_objects":457,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11683,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'11953","reported_seq":16132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:23.981036+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:23.981036+0000","last_peered":"2024-09-10T20:30:23.981036+0000","last_clean":"2024-09-10T20:30:23.981036+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:23.981036+0000","last_undegraded":"2024-09-10T20:30:23.981036+0000","last_fullsized":"2024-09-10T20:30:23.981036+0000","mapping_epoch":37,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3710976,"num_objects":453,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11691","reported_seq":12021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.384529+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:25.384529+0000","last_peered":"2024-09-10T20:30:25.384529+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:25.384529+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3912864,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11647","reported_seq":11877,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.325475+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:25.325475+0000","last_peered":"2024-09-10T20:30:25.325475+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:25.325475+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3694592,"num_objects":451,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11647,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11651","reported_seq":13367,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:23.722721+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:23.722721+0000","last_peered":"2024-09-10T20:30:23.722721+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:23.722721+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3429536,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":813,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11827","reported_seq":12002,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:23.729797+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:23.729797+0000","last_peered":"2024-09-10T20:30:23.729797+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:23.729797+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814560,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11825,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11628","reported_seq":15610,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:25.220908+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:25.220908+0000","last_peered":"2024-09-10T20:30:25.220908+0000","last_clean":"2024-09-10T20:30:25.220908+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:25.220908+0000","last_undegraded":"2024-09-10T20:30:25.220908+0000","last_fullsized":"2024-09-10T20:30:25.220908+0000","mapping_epoch":33,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3054160,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11676","reported_seq":15655,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:21.860294+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:21.860294+0000","last_peered":"2024-09-10T20:30:21.860294+0000","last_clean":"2024-09-10T20:30:21.860294+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:21.860294+0000","last_undegraded":"2024-09-10T20:30:21.860294+0000","last_fullsized":"2024-09-10T20:30:21.860294+0000","mapping_epoch":39,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3817472,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4314,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11672","reported_seq":11955,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.324313+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:25.324313+0000","last_peered":"2024-09-10T20:30:25.324313+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:25.324313+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3608304,"num_objects":441,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11713","reported_seq":12019,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.485856+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:25.485856+0000","last_peered":"2024-09-10T20:30:25.485856+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:25.485856+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701328,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11399","reported_seq":11592,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.886357+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:21.886357+0000","last_peered":"2024-09-10T20:30:21.886357+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:21.886357+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3537488,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11398,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11660","reported_seq":15719,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:24.061025+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:24.061025+0000","last_peered":"2024-09-10T20:30:24.061025+0000","last_clean":"2024-09-10T20:30:24.061025+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:24.061025+0000","last_undegraded":"2024-09-10T20:30:24.061025+0000","last_fullsized":"2024-09-10T20:30:24.061025+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'11987","reported_seq":12202,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:21.884020+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:21.884020+0000","last_peered":"2024-09-10T20:30:21.884020+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:21.884020+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781792,"num_objects":462,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11676","reported_seq":15780,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:23.885068+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:23.885068+0000","last_peered":"2024-09-10T20:30:23.885068+0000","last_clean":"2024-09-10T20:30:23.885068+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:23.885068+0000","last_undegraded":"2024-09-10T20:30:23.885068+0000","last_fullsized":"2024-09-10T20:30:23.885068+0000","mapping_epoch":41,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3661824,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2549,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":57592520,"num_objects":7034,"num_object_clones":0,"num_object_copies":28136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7034,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147136,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116002816,"data_stored":116002816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694976,"kb_used_data":27440,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93676864,"statfs":{"total":96636764160,"available":95925108736,"internally_reserved":0,"allocated":28098560,"data_stored":27866515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,6,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702200,"kb_used_data":31012,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93669640,"statfs":{"total":96636764160,"available":95917711360,"internally_reserved":0,"allocated":31756288,"data_stored":31524243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019003,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562112,"kb_used_data":29232,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93809728,"statfs":{"total":96636764160,"available":96061161472,"internally_reserved":0,"allocated":29933568,"data_stored":29701523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545648338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,11,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700444,"kb_used_data":28812,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93671396,"statfs":{"total":96636764160,"available":95919509504,"internally_reserved":0,"allocated":29503488,"data_stored":29271443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29048832,"data_stored":29048832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28704768,"data_stored":28704768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30658560,"data_stored":30658560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27590656,"data_stored":27590656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:30.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:31.250 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:31.276 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-10T20:30:30.130161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":52349640,"num_objects":6394,"num_object_clones":0,"num_object_copies":25576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147776,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160573,"ondisk_log_size":160573,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2642460,"kb_used_data":103380,"kb_used_omap":14,"kb_used_meta":2538993,"kb_avail":374844900,"statfs":{"total":386547056640,"available":383841177600,"internally_reserved":0,"allocated":105861120,"data_stored":104932940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2599929398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,9,3,11,25,25,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19283968,"num_objects":-2354,"num_object_clones":0,"num_object_copies":-9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004325"},"pg_stats":[{"pgid":"1.f","version":"44'11722","reported_seq":12013,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.261291+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:25.261291+0000","last_peered":"2024-09-10T20:30:25.261291+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:25.261291+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":400,"num_object_clones":0,"num_object_copies":1600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11722,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11902","reported_seq":12043,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:28.907716+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:28.907716+0000","last_peered":"2024-09-10T20:30:28.907716+0000","last_clean":"2024-09-10T20:30:28.907716+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:28.907716+0000","last_undegraded":"2024-09-10T20:30:28.907716+0000","last_fullsized":"2024-09-10T20:30:28.907716+0000","mapping_epoch":35,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2867200,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11750","reported_seq":11963,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:26.889624+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:26.889624+0000","last_peered":"2024-09-10T20:30:26.889624+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:26.889624+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193448,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11749,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12031","reported_seq":16210,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:29.078158+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:29.078158+0000","last_peered":"2024-09-10T20:30:29.078158+0000","last_clean":"2024-09-10T20:30:29.078158+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:29.078158+0000","last_undegraded":"2024-09-10T20:30:29.078158+0000","last_fullsized":"2024-09-10T20:30:29.078158+0000","mapping_epoch":37,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3072000,"num_objects":375,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11691","reported_seq":12021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.384529+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:25.384529+0000","last_peered":"2024-09-10T20:30:25.384529+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:25.384529+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3912864,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11647","reported_seq":11877,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.325475+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:25.325475+0000","last_peered":"2024-09-10T20:30:25.325475+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:25.325475+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3694592,"num_objects":451,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11647,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11704","reported_seq":13545,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:28.830411+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:28.830411+0000","last_peered":"2024-09-10T20:30:28.830411+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:28.830411+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2995360,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-715}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11908","reported_seq":12086,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:28.662551+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:28.662551+0000","last_peered":"2024-09-10T20:30:28.662551+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:28.662551+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3151008,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11628","reported_seq":15610,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:25.220908+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:25.220908+0000","last_peered":"2024-09-10T20:30:25.220908+0000","last_clean":"2024-09-10T20:30:25.220908+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:25.220908+0000","last_undegraded":"2024-09-10T20:30:25.220908+0000","last_fullsized":"2024-09-10T20:30:25.220908+0000","mapping_epoch":33,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3054160,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11742","reported_seq":15722,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:26.877460+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:26.877460+0000","last_peered":"2024-09-10T20:30:26.877460+0000","last_clean":"2024-09-10T20:30:26.877460+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:26.877460+0000","last_undegraded":"2024-09-10T20:30:26.877460+0000","last_fullsized":"2024-09-10T20:30:26.877460+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":400,"num_object_clones":0,"num_object_copies":1600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4380,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11672","reported_seq":11955,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.324313+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:25.324313+0000","last_peered":"2024-09-10T20:30:25.324313+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:25.324313+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3608304,"num_objects":441,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11713","reported_seq":12019,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.485856+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:25.485856+0000","last_peered":"2024-09-10T20:30:25.485856+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:25.485856+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701328,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11451","reported_seq":11644,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:26.953617+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:26.953617+0000","last_peered":"2024-09-10T20:30:26.953617+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:26.953617+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3111504,"num_objects":380,"num_object_clones":0,"num_object_copies":1520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11730","reported_seq":15790,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:28.995803+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:28.995803+0000","last_peered":"2024-09-10T20:30:28.995803+0000","last_clean":"2024-09-10T20:30:28.995803+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:28.995803+0000","last_undegraded":"2024-09-10T20:30:28.995803+0000","last_fullsized":"2024-09-10T20:30:28.995803+0000","mapping_epoch":43,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2965504,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":682,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12043","reported_seq":12258,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:26.820132+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:26.820132+0000","last_peered":"2024-09-10T20:30:26.820132+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:26.820132+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3323040,"num_objects":406,"num_object_clones":0,"num_object_copies":1624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12041,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1502}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11739","reported_seq":15841,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:29.175668+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:29.175668+0000","last_peered":"2024-09-10T20:30:29.175668+0000","last_clean":"2024-09-10T20:30:29.175668+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:29.175668+0000","last_undegraded":"2024-09-10T20:30:29.175668+0000","last_fullsized":"2024-09-10T20:30:29.175668+0000","mapping_epoch":41,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":384,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":52349640,"num_objects":6394,"num_object_clones":0,"num_object_copies":25576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147776,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":103280640,"data_stored":103280640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160573,"ondisk_log_size":160573,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694976,"kb_used_data":27440,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93676864,"statfs":{"total":96636764160,"available":95925108736,"internally_reserved":0,"allocated":28098560,"data_stored":27866515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,6,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702248,"kb_used_data":26904,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":93669592,"statfs":{"total":96636764160,"available":95917662208,"internally_reserved":0,"allocated":27549696,"data_stored":27317651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,0,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561444,"kb_used_data":24404,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93810396,"statfs":{"total":96636764160,"available":96061845504,"internally_reserved":0,"allocated":24989696,"data_stored":24757651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,14,10,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683792,"kb_used_data":24632,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93688048,"statfs":{"total":96636764160,"available":95936561152,"internally_reserved":0,"allocated":25223168,"data_stored":24991123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24821760,"data_stored":24821760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24518656,"data_stored":24518656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26349568,"data_stored":26349568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27590656,"data_stored":27590656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:31.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:31.859 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:31.880 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-10T20:30:30.130161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":52349640,"num_objects":6394,"num_object_clones":0,"num_object_copies":25576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147776,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160573,"ondisk_log_size":160573,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2642460,"kb_used_data":103380,"kb_used_omap":14,"kb_used_meta":2538993,"kb_avail":374844900,"statfs":{"total":386547056640,"available":383841177600,"internally_reserved":0,"allocated":105861120,"data_stored":104932940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2599929398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,9,3,11,25,25,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19283968,"num_objects":-2354,"num_object_clones":0,"num_object_copies":-9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004325"},"pg_stats":[{"pgid":"1.f","version":"44'11722","reported_seq":12013,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.261291+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:25.261291+0000","last_peered":"2024-09-10T20:30:25.261291+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:25.261291+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":400,"num_object_clones":0,"num_object_copies":1600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11722,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11902","reported_seq":12043,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:28.907716+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:28.907716+0000","last_peered":"2024-09-10T20:30:28.907716+0000","last_clean":"2024-09-10T20:30:28.907716+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:28.907716+0000","last_undegraded":"2024-09-10T20:30:28.907716+0000","last_fullsized":"2024-09-10T20:30:28.907716+0000","mapping_epoch":35,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2867200,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11750","reported_seq":11963,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:26.889624+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:26.889624+0000","last_peered":"2024-09-10T20:30:26.889624+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:26.889624+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193448,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11749,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12031","reported_seq":16210,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:29.078158+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:29.078158+0000","last_peered":"2024-09-10T20:30:29.078158+0000","last_clean":"2024-09-10T20:30:29.078158+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:29.078158+0000","last_undegraded":"2024-09-10T20:30:29.078158+0000","last_fullsized":"2024-09-10T20:30:29.078158+0000","mapping_epoch":37,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3072000,"num_objects":375,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11691","reported_seq":12021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.384529+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:25.384529+0000","last_peered":"2024-09-10T20:30:25.384529+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:25.384529+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3912864,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11647","reported_seq":11877,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.325475+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:25.325475+0000","last_peered":"2024-09-10T20:30:25.325475+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:25.325475+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3694592,"num_objects":451,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11647,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11704","reported_seq":13545,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:28.830411+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:28.830411+0000","last_peered":"2024-09-10T20:30:28.830411+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:28.830411+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2995360,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-715}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11908","reported_seq":12086,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:28.662551+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:28.662551+0000","last_peered":"2024-09-10T20:30:28.662551+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:28.662551+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3151008,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11628","reported_seq":15610,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:25.220908+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:25.220908+0000","last_peered":"2024-09-10T20:30:25.220908+0000","last_clean":"2024-09-10T20:30:25.220908+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:25.220908+0000","last_undegraded":"2024-09-10T20:30:25.220908+0000","last_fullsized":"2024-09-10T20:30:25.220908+0000","mapping_epoch":33,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3054160,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11742","reported_seq":15722,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:26.877460+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:26.877460+0000","last_peered":"2024-09-10T20:30:26.877460+0000","last_clean":"2024-09-10T20:30:26.877460+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:26.877460+0000","last_undegraded":"2024-09-10T20:30:26.877460+0000","last_fullsized":"2024-09-10T20:30:26.877460+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":400,"num_object_clones":0,"num_object_copies":1600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4380,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11672","reported_seq":11955,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.324313+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:25.324313+0000","last_peered":"2024-09-10T20:30:25.324313+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:25.324313+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3608304,"num_objects":441,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11713","reported_seq":12019,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:25.485856+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:25.485856+0000","last_peered":"2024-09-10T20:30:25.485856+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:25.485856+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701328,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11451","reported_seq":11644,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:26.953617+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:26.953617+0000","last_peered":"2024-09-10T20:30:26.953617+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:26.953617+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3111504,"num_objects":380,"num_object_clones":0,"num_object_copies":1520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11730","reported_seq":15790,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:28.995803+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:28.995803+0000","last_peered":"2024-09-10T20:30:28.995803+0000","last_clean":"2024-09-10T20:30:28.995803+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:28.995803+0000","last_undegraded":"2024-09-10T20:30:28.995803+0000","last_fullsized":"2024-09-10T20:30:28.995803+0000","mapping_epoch":43,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2965504,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":682,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12043","reported_seq":12258,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:26.820132+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:26.820132+0000","last_peered":"2024-09-10T20:30:26.820132+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:26.820132+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3323040,"num_objects":406,"num_object_clones":0,"num_object_copies":1624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12041,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1502}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11739","reported_seq":15841,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:29.175668+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:29.175668+0000","last_peered":"2024-09-10T20:30:29.175668+0000","last_clean":"2024-09-10T20:30:29.175668+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:29.175668+0000","last_undegraded":"2024-09-10T20:30:29.175668+0000","last_fullsized":"2024-09-10T20:30:29.175668+0000","mapping_epoch":41,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":384,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":52349640,"num_objects":6394,"num_object_clones":0,"num_object_copies":25576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147776,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":103280640,"data_stored":103280640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160573,"ondisk_log_size":160573,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694976,"kb_used_data":27440,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93676864,"statfs":{"total":96636764160,"available":95925108736,"internally_reserved":0,"allocated":28098560,"data_stored":27866515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,6,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702248,"kb_used_data":26904,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":93669592,"statfs":{"total":96636764160,"available":95917662208,"internally_reserved":0,"allocated":27549696,"data_stored":27317651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,0,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561444,"kb_used_data":24404,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93810396,"statfs":{"total":96636764160,"available":96061845504,"internally_reserved":0,"allocated":24989696,"data_stored":24757651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,14,10,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683792,"kb_used_data":24632,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93688048,"statfs":{"total":96636764160,"available":95936561152,"internally_reserved":0,"allocated":25223168,"data_stored":24991123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24821760,"data_stored":24821760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24518656,"data_stored":24518656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26349568,"data_stored":26349568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27590656,"data_stored":27590656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:31.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:31.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11722", 'reported_seq': 12013, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:25.261291+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:25.261291+0000', 'last_peered': '2024-09-10T20:30:25.261291+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:25.261291+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3276800, 'num_objects': 400, 'num_object_clones': 0, 'num_object_copies': 1600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11722, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1455}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:31.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:31.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11750", 'reported_seq': 11963, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:26.889624+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:26.889624+0000', 'last_peered': '2024-09-10T20:30:26.889624+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:26.889624+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3193448, 'num_objects': 391, 'num_object_clones': 0, 'num_object_copies': 1564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 391, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11749, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1517}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:31.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:31.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11691", 'reported_seq': 12021, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:25.384529+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:25.384529+0000', 'last_peered': '2024-09-10T20:30:25.384529+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:25.384529+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3912864, 'num_objects': 478, 'num_object_clones': 0, 'num_object_copies': 1912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11689, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1377}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:31.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11647", 'reported_seq': 11877, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:25.325475+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:25.325475+0000', 'last_peered': '2024-09-10T20:30:25.325475+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:25.325475+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3694592, 'num_objects': 451, 'num_object_clones': 0, 'num_object_copies': 1804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11647, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1406}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11704", 'reported_seq': 13545, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:28.830411+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:28.830411+0000', 'last_peered': '2024-09-10T20:30:28.830411+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:28.830411+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2995360, 'num_objects': 366, 'num_object_clones': 0, 'num_object_copies': 1464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11702, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 876, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -715}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11908", 'reported_seq': 12086, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:28.662551+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:28.662551+0000', 'last_peered': '2024-09-10T20:30:28.662551+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:28.662551+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3151008, 'num_objects': 385, 'num_object_clones': 0, 'num_object_copies': 1540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11906, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11672", 'reported_seq': 11955, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:25.324313+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:25.324313+0000', 'last_peered': '2024-09-10T20:30:25.324313+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:25.324313+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3608304, 'num_objects': 441, 'num_object_clones': 0, 'num_object_copies': 1764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11669, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1415}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:31.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11713", 'reported_seq': 12019, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:25.485856+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:25.485856+0000', 'last_peered': '2024-09-10T20:30:25.485856+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:25.485856+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3701328, 'num_objects': 452, 'num_object_clones': 0, 'num_object_copies': 1808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11712, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:31.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:31.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11451", 'reported_seq': 11644, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:26.953617+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:26.953617+0000', 'last_peered': '2024-09-10T20:30:26.953617+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:26.953617+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3111504, 'num_objects': 380, 'num_object_clones': 0, 'num_object_copies': 1520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11450, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:31.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:31.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12043", 'reported_seq': 12258, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:26.820132+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:26.820132+0000', 'last_peered': '2024-09-10T20:30:26.820132+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:26.820132+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3323040, 'num_objects': 406, 'num_object_clones': 0, 'num_object_copies': 1624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12041, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:31.884 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:32.588 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:32.588 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18095/25576 objects degraded (70.750%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":721,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6394,"data_bytes":52349640,"bytes_used":2705879040,"bytes_avail":383841177600,"bytes_total":386547056640,"degraded_objects":18095,"degraded_total":25576,"degraded_ratio":0.70749921801689086,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:32.613 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:32.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:33.234 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:33.234 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:33.256 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-10T20:30:32.130988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46942920,"num_objects":5734,"num_object_clones":0,"num_object_copies":22936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148436,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643196,"kb_used_data":94388,"kb_used_omap":14,"kb_used_meta":2548721,"kb_avail":374844164,"statfs":{"total":386547056640,"available":383840423936,"internally_reserved":0,"allocated":96653312,"data_stored":95725132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2609890870},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,9,6,25,22,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21807104,"num_objects":-2662,"num_object_clones":0,"num_object_copies":-10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004713"},"pg_stats":[{"pgid":"1.f","version":"44'11777","reported_seq":12067,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:29.978580+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:29.978580+0000","last_peered":"2024-09-10T20:30:29.978580+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:29.978580+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2826240,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11902","reported_seq":12043,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:28.907716+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:28.907716+0000","last_peered":"2024-09-10T20:30:28.907716+0000","last_clean":"2024-09-10T20:30:28.907716+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:28.907716+0000","last_undegraded":"2024-09-10T20:30:28.907716+0000","last_fullsized":"2024-09-10T20:30:28.907716+0000","mapping_epoch":35,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2867200,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11824","reported_seq":12037,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.857547+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:31.857547+0000","last_peered":"2024-09-10T20:30:31.857547+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:31.857547+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2587240,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11823,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1591}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12031","reported_seq":16210,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:29.078158+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:29.078158+0000","last_peered":"2024-09-10T20:30:29.078158+0000","last_clean":"2024-09-10T20:30:29.078158+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:29.078158+0000","last_undegraded":"2024-09-10T20:30:29.078158+0000","last_fullsized":"2024-09-10T20:30:29.078158+0000","mapping_epoch":37,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3072000,"num_objects":375,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11769","reported_seq":12099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:30.448764+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:30.448764+0000","last_peered":"2024-09-10T20:30:30.448764+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:30.448764+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3273888,"num_objects":400,"num_object_clones":0,"num_object_copies":1600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11711","reported_seq":11942,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:30.348440+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:30.348440+0000","last_peered":"2024-09-10T20:30:30.348440+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:30.348440+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3170304,"num_objects":387,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11704","reported_seq":13545,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:28.830411+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:28.830411+0000","last_peered":"2024-09-10T20:30:28.830411+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:28.830411+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2995360,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-715}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11908","reported_seq":12086,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:28.662551+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:28.662551+0000","last_peered":"2024-09-10T20:30:28.662551+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:28.662551+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3151008,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11682","reported_seq":15664,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:30.252154+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:30.252154+0000","last_peered":"2024-09-10T20:30:30.252154+0000","last_clean":"2024-09-10T20:30:30.252154+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:30.252154+0000","last_undegraded":"2024-09-10T20:30:30.252154+0000","last_fullsized":"2024-09-10T20:30:30.252154+0000","mapping_epoch":33,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2611792,"num_objects":319,"num_object_clones":0,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8235,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11797","reported_seq":15776,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:32.021918+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:32.021918+0000","last_peered":"2024-09-10T20:30:32.021918+0000","last_clean":"2024-09-10T20:30:32.021918+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:32.021918+0000","last_undegraded":"2024-09-10T20:30:32.021918+0000","last_fullsized":"2024-09-10T20:30:32.021918+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2826240,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11748","reported_seq":12031,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:30.491490+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:30.491490+0000","last_peered":"2024-09-10T20:30:30.491490+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:30.491490+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2985712,"num_objects":365,"num_object_clones":0,"num_object_copies":1460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11789","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:30.359015+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:30.359015+0000","last_peered":"2024-09-10T20:30:30.359015+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:30.359015+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3078736,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1484}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11517","reported_seq":11710,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.834465+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:31.834465+0000","last_peered":"2024-09-10T20:30:31.834465+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:31.834465+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2570832,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1589}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11730","reported_seq":15790,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:28.995803+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:28.995803+0000","last_peered":"2024-09-10T20:30:28.995803+0000","last_clean":"2024-09-10T20:30:28.995803+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:28.995803+0000","last_undegraded":"2024-09-10T20:30:28.995803+0000","last_fullsized":"2024-09-10T20:30:28.995803+0000","mapping_epoch":43,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2965504,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":682,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12105","reported_seq":12320,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.744374+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:31.744374+0000","last_peered":"2024-09-10T20:30:31.744374+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:31.744374+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2815136,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1564}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11739","reported_seq":15841,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:29.175668+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:29.175668+0000","last_peered":"2024-09-10T20:30:29.175668+0000","last_clean":"2024-09-10T20:30:29.175668+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:29.175668+0000","last_undegraded":"2024-09-10T20:30:29.175668+0000","last_fullsized":"2024-09-10T20:30:29.175668+0000","mapping_epoch":41,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":384,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":46942920,"num_objects":5734,"num_object_clones":0,"num_object_copies":22936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148436,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94785536,"data_stored":94785536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695036,"kb_used_data":23340,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93676804,"statfs":{"total":96636764160,"available":95925047296,"internally_reserved":0,"allocated":23900160,"data_stored":23668115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,8,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702924,"kb_used_data":22012,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93668916,"statfs":{"total":96636764160,"available":95916969984,"internally_reserved":0,"allocated":22540288,"data_stored":22308243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561444,"kb_used_data":24404,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93810396,"statfs":{"total":96636764160,"available":96061845504,"internally_reserved":0,"allocated":24989696,"data_stored":24757651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,14,10,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683792,"kb_used_data":24632,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93688048,"statfs":{"total":96636764160,"available":95936561152,"internally_reserved":0,"allocated":25223168,"data_stored":24991123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24821760,"data_stored":24821760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24518656,"data_stored":24518656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22097920,"data_stored":22097920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23347200,"data_stored":23347200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:36.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:36.893 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:36.894 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:36.919 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-10T20:30:36.132314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40569544,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149214,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681156,"kb_used_data":82352,"kb_used_omap":14,"kb_used_meta":2598705,"kb_avail":374806204,"statfs":{"total":386547056640,"available":383801552896,"internally_reserved":0,"allocated":84328448,"data_stored":83400268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2661074486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,10,11,8,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22380544,"num_objects":-2732,"num_object_clones":0,"num_object_copies":-10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004510"},"pg_stats":[{"pgid":"1.f","version":"44'11844","reported_seq":12135,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.482846+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:35.482846+0000","last_peered":"2024-09-10T20:30:35.482846+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:35.482846+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2277376,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11966","reported_seq":12107,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.180810+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:34.180810+0000","last_peered":"2024-09-10T20:30:34.180810+0000","last_clean":"2024-09-10T20:30:34.180810+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:34.180810+0000","last_undegraded":"2024-09-10T20:30:34.180810+0000","last_fullsized":"2024-09-10T20:30:34.180810+0000","mapping_epoch":35,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2342912,"num_objects":286,"num_object_clones":0,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11824","reported_seq":12037,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.857547+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:31.857547+0000","last_peered":"2024-09-10T20:30:31.857547+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:31.857547+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2587240,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11823,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1591}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12097","reported_seq":16276,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.057925+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:34.057925+0000","last_peered":"2024-09-10T20:30:34.057925+0000","last_clean":"2024-09-10T20:30:34.057925+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:34.057925+0000","last_undegraded":"2024-09-10T20:30:34.057925+0000","last_fullsized":"2024-09-10T20:30:34.057925+0000","mapping_epoch":37,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2531328,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6357,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11849","reported_seq":12179,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.435958+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:35.435958+0000","last_peered":"2024-09-10T20:30:35.435958+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:35.435958+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2618528,"num_objects":320,"num_object_clones":0,"num_object_copies":1280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1535}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11777","reported_seq":12008,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.296939+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:35.296939+0000","last_peered":"2024-09-10T20:30:35.296939+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:35.296939+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2629632,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11758","reported_seq":13712,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:33.898581+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:33.898581+0000","last_peered":"2024-09-10T20:30:33.898581+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:33.898581+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2552992,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-713}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11974","reported_seq":12152,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:33.862879+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:33.862879+0000","last_peered":"2024-09-10T20:30:33.862879+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:33.862879+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2610336,"num_objects":319,"num_object_clones":0,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11972,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11742","reported_seq":15724,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:35.160482+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:35.160482+0000","last_peered":"2024-09-10T20:30:35.160482+0000","last_clean":"2024-09-10T20:30:35.160482+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:35.160482+0000","last_undegraded":"2024-09-10T20:30:35.160482+0000","last_fullsized":"2024-09-10T20:30:35.160482+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2120272,"num_objects":259,"num_object_clones":0,"num_object_copies":1036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11797","reported_seq":15776,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:32.021918+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:32.021918+0000","last_peered":"2024-09-10T20:30:32.021918+0000","last_clean":"2024-09-10T20:30:32.021918+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:32.021918+0000","last_undegraded":"2024-09-10T20:30:32.021918+0000","last_fullsized":"2024-09-10T20:30:32.021918+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2826240,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11832","reported_seq":12115,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.366625+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:35.366625+0000","last_peered":"2024-09-10T20:30:35.366625+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:35.366625+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2297584,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11842","reported_seq":12150,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.402307+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:35.402307+0000","last_peered":"2024-09-10T20:30:35.402307+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:35.402307+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2644560,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11517","reported_seq":11710,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.834465+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:31.834465+0000","last_peered":"2024-09-10T20:30:31.834465+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:31.834465+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2570832,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1589}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11789","reported_seq":15848,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.236655+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:34.236655+0000","last_peered":"2024-09-10T20:30:34.236655+0000","last_clean":"2024-09-10T20:30:34.236655+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:34.236655+0000","last_undegraded":"2024-09-10T20:30:34.236655+0000","last_fullsized":"2024-09-10T20:30:34.236655+0000","mapping_epoch":43,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2482176,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12105","reported_seq":12320,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.744374+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:31.744374+0000","last_peered":"2024-09-10T20:30:31.744374+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:31.744374+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2815136,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1564}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11798","reported_seq":15902,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.261242+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:34.261242+0000","last_peered":"2024-09-10T20:30:34.261242+0000","last_clean":"2024-09-10T20:30:34.261242+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:34.261242+0000","last_undegraded":"2024-09-10T20:30:34.261242+0000","last_fullsized":"2024-09-10T20:30:34.261242+0000","mapping_epoch":41,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2662400,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2671,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40569544,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149214,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":82206720,"data_stored":82206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696632,"kb_used_data":19348,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93675208,"statfs":{"total":96636764160,"available":95923412992,"internally_reserved":0,"allocated":19812352,"data_stored":19580307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702924,"kb_used_data":22012,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93668916,"statfs":{"total":96636764160,"available":95916969984,"internally_reserved":0,"allocated":22540288,"data_stored":22308243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579644,"kb_used_data":20396,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93792196,"statfs":{"total":96636764160,"available":96043208704,"internally_reserved":0,"allocated":20885504,"data_stored":20653459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701956,"kb_used_data":20596,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93669884,"statfs":{"total":96636764160,"available":95917961216,"internally_reserved":0,"allocated":21090304,"data_stored":20858259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20586496,"data_stored":20586496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20340736,"data_stored":20340736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22097920,"data_stored":22097920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19181568,"data_stored":19181568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:36.920 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:37.562 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:37.563 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:37.588 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-10T20:30:36.132314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40569544,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149214,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681156,"kb_used_data":82352,"kb_used_omap":14,"kb_used_meta":2598705,"kb_avail":374806204,"statfs":{"total":386547056640,"available":383801552896,"internally_reserved":0,"allocated":84328448,"data_stored":83400268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2661074486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,10,11,8,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22380544,"num_objects":-2732,"num_object_clones":0,"num_object_copies":-10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004510"},"pg_stats":[{"pgid":"1.f","version":"44'11844","reported_seq":12135,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.482846+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:35.482846+0000","last_peered":"2024-09-10T20:30:35.482846+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:35.482846+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2277376,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11966","reported_seq":12107,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.180810+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:34.180810+0000","last_peered":"2024-09-10T20:30:34.180810+0000","last_clean":"2024-09-10T20:30:34.180810+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:34.180810+0000","last_undegraded":"2024-09-10T20:30:34.180810+0000","last_fullsized":"2024-09-10T20:30:34.180810+0000","mapping_epoch":35,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2342912,"num_objects":286,"num_object_clones":0,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11824","reported_seq":12037,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.857547+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:31.857547+0000","last_peered":"2024-09-10T20:30:31.857547+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:31.857547+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2587240,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11823,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1591}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12097","reported_seq":16276,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.057925+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:34.057925+0000","last_peered":"2024-09-10T20:30:34.057925+0000","last_clean":"2024-09-10T20:30:34.057925+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:34.057925+0000","last_undegraded":"2024-09-10T20:30:34.057925+0000","last_fullsized":"2024-09-10T20:30:34.057925+0000","mapping_epoch":37,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2531328,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6357,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11849","reported_seq":12179,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.435958+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:35.435958+0000","last_peered":"2024-09-10T20:30:35.435958+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:35.435958+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2618528,"num_objects":320,"num_object_clones":0,"num_object_copies":1280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1535}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11777","reported_seq":12008,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.296939+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:35.296939+0000","last_peered":"2024-09-10T20:30:35.296939+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:35.296939+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2629632,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11758","reported_seq":13712,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:33.898581+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:33.898581+0000","last_peered":"2024-09-10T20:30:33.898581+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:33.898581+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2552992,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-713}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11974","reported_seq":12152,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:33.862879+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:33.862879+0000","last_peered":"2024-09-10T20:30:33.862879+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:33.862879+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2610336,"num_objects":319,"num_object_clones":0,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11972,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11742","reported_seq":15724,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:35.160482+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:35.160482+0000","last_peered":"2024-09-10T20:30:35.160482+0000","last_clean":"2024-09-10T20:30:35.160482+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:35.160482+0000","last_undegraded":"2024-09-10T20:30:35.160482+0000","last_fullsized":"2024-09-10T20:30:35.160482+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2120272,"num_objects":259,"num_object_clones":0,"num_object_copies":1036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11797","reported_seq":15776,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:32.021918+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:32.021918+0000","last_peered":"2024-09-10T20:30:32.021918+0000","last_clean":"2024-09-10T20:30:32.021918+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:32.021918+0000","last_undegraded":"2024-09-10T20:30:32.021918+0000","last_fullsized":"2024-09-10T20:30:32.021918+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2826240,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11832","reported_seq":12115,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.366625+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:35.366625+0000","last_peered":"2024-09-10T20:30:35.366625+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:35.366625+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2297584,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11842","reported_seq":12150,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.402307+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:35.402307+0000","last_peered":"2024-09-10T20:30:35.402307+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:35.402307+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2644560,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11517","reported_seq":11710,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.834465+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:31.834465+0000","last_peered":"2024-09-10T20:30:31.834465+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:31.834465+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2570832,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1589}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11789","reported_seq":15848,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.236655+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:34.236655+0000","last_peered":"2024-09-10T20:30:34.236655+0000","last_clean":"2024-09-10T20:30:34.236655+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:34.236655+0000","last_undegraded":"2024-09-10T20:30:34.236655+0000","last_fullsized":"2024-09-10T20:30:34.236655+0000","mapping_epoch":43,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2482176,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12105","reported_seq":12320,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:31.744374+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:31.744374+0000","last_peered":"2024-09-10T20:30:31.744374+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:31.744374+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2815136,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1564}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11798","reported_seq":15902,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.261242+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:34.261242+0000","last_peered":"2024-09-10T20:30:34.261242+0000","last_clean":"2024-09-10T20:30:34.261242+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:34.261242+0000","last_undegraded":"2024-09-10T20:30:34.261242+0000","last_fullsized":"2024-09-10T20:30:34.261242+0000","mapping_epoch":41,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2662400,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2671,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40569544,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149214,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":82206720,"data_stored":82206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696632,"kb_used_data":19348,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93675208,"statfs":{"total":96636764160,"available":95923412992,"internally_reserved":0,"allocated":19812352,"data_stored":19580307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702924,"kb_used_data":22012,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93668916,"statfs":{"total":96636764160,"available":95916969984,"internally_reserved":0,"allocated":22540288,"data_stored":22308243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579644,"kb_used_data":20396,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93792196,"statfs":{"total":96636764160,"available":96043208704,"internally_reserved":0,"allocated":20885504,"data_stored":20653459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701956,"kb_used_data":20596,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93669884,"statfs":{"total":96636764160,"available":95917961216,"internally_reserved":0,"allocated":21090304,"data_stored":20858259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20586496,"data_stored":20586496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20340736,"data_stored":20340736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22097920,"data_stored":22097920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19181568,"data_stored":19181568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11844", 'reported_seq': 12135, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:35.482846+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:35.482846+0000', 'last_peered': '2024-09-10T20:30:35.482846+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:35.482846+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2277376, 'num_objects': 278, 'num_object_clones': 0, 'num_object_copies': 1112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11844, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1577}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11824", 'reported_seq': 12037, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:31.857547+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:31.857547+0000', 'last_peered': '2024-09-10T20:30:31.857547+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:31.857547+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2587240, 'num_objects': 317, 'num_object_clones': 0, 'num_object_copies': 1268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 317, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11823, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1591}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11849", 'reported_seq': 12179, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:35.435958+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:35.435958+0000', 'last_peered': '2024-09-10T20:30:35.435958+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:35.435958+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2618528, 'num_objects': 320, 'num_object_clones': 0, 'num_object_copies': 1280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11847, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1535}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11777", 'reported_seq': 12008, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:35.296939+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:35.296939+0000', 'last_peered': '2024-09-10T20:30:35.296939+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:35.296939+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2629632, 'num_objects': 321, 'num_object_clones': 0, 'num_object_copies': 1284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1536}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:37.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11758", 'reported_seq': 13712, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:33.898581+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:33.898581+0000', 'last_peered': '2024-09-10T20:30:33.898581+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:33.898581+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2552992, 'num_objects': 312, 'num_object_clones': 0, 'num_object_copies': 1248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11756, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 932, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -713}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'11974", 'reported_seq': 12152, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:33.862879+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:33.862879+0000', 'last_peered': '2024-09-10T20:30:33.862879+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:33.862879+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2610336, 'num_objects': 319, 'num_object_clones': 0, 'num_object_copies': 1276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11972, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1681}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11832", 'reported_seq': 12115, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:35.366625+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:35.366625+0000', 'last_peered': '2024-09-10T20:30:35.366625+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:35.366625+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2297584, 'num_objects': 281, 'num_object_clones': 0, 'num_object_copies': 1124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1575}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11842", 'reported_seq': 12150, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:35.402307+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:35.402307+0000', 'last_peered': '2024-09-10T20:30:35.402307+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:35.402307+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2644560, 'num_objects': 323, 'num_object_clones': 0, 'num_object_copies': 1292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11841, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11517", 'reported_seq': 11710, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:31.834465+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:31.834465+0000', 'last_peered': '2024-09-10T20:30:31.834465+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:31.834465+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2570832, 'num_objects': 314, 'num_object_clones': 0, 'num_object_copies': 1256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 314, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11516, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1589}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:37.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12105", 'reported_seq': 12320, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:31.744374+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:31.744374+0000', 'last_peered': '2024-09-10T20:30:31.744374+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:31.744374+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2815136, 'num_objects': 344, 'num_object_clones': 0, 'num_object_copies': 1376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12103, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1564}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:37.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:38.352 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:38.352 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18039/19824 objects degraded (90.996%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":727,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4956,"data_bytes":40569544,"bytes_used":2745503744,"bytes_avail":383801552896,"bytes_total":386547056640,"degraded_objects":18039,"degraded_total":19824,"degraded_ratio":0.90995762711864403,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":227},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:38.371 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:38.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:39.036 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:39.037 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:39.073 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-10T20:30:38.133064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38423240,"num_objects":4694,"num_object_clones":0,"num_object_copies":18776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149476,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161073,"ondisk_log_size":161073,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681232,"kb_used_data":78264,"kb_used_omap":14,"kb_used_meta":2602865,"kb_avail":374806128,"statfs":{"total":386547056640,"available":383801475072,"internally_reserved":0,"allocated":80142336,"data_stored":79214156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2665334326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,12,10,9,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19169280,"num_objects":-2340,"num_object_clones":0,"num_object_copies":-9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004445"},"pg_stats":[{"pgid":"1.f","version":"44'11844","reported_seq":12135,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.482846+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:35.482846+0000","last_peered":"2024-09-10T20:30:35.482846+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:35.482846+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2277376,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'11966","reported_seq":12107,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.180810+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:34.180810+0000","last_peered":"2024-09-10T20:30:34.180810+0000","last_clean":"2024-09-10T20:30:34.180810+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:34.180810+0000","last_undegraded":"2024-09-10T20:30:34.180810+0000","last_fullsized":"2024-09-10T20:30:34.180810+0000","mapping_epoch":35,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2342912,"num_objects":286,"num_object_clones":0,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11883","reported_seq":12096,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.048741+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:37.048741+0000","last_peered":"2024-09-10T20:30:37.048741+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:37.048741+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2103912,"num_objects":258,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11882,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12097","reported_seq":16276,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.057925+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:34.057925+0000","last_peered":"2024-09-10T20:30:34.057925+0000","last_clean":"2024-09-10T20:30:34.057925+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:34.057925+0000","last_undegraded":"2024-09-10T20:30:34.057925+0000","last_fullsized":"2024-09-10T20:30:34.057925+0000","mapping_epoch":37,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2531328,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6357,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11849","reported_seq":12179,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.435958+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:35.435958+0000","last_peered":"2024-09-10T20:30:35.435958+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:35.435958+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2618528,"num_objects":320,"num_object_clones":0,"num_object_copies":1280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1535}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11777","reported_seq":12008,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.296939+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:35.296939+0000","last_peered":"2024-09-10T20:30:35.296939+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:35.296939+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2629632,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11758","reported_seq":13712,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:33.898581+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:33.898581+0000","last_peered":"2024-09-10T20:30:33.898581+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:33.898581+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2552992,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-713}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'11974","reported_seq":12152,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:33.862879+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:33.862879+0000","last_peered":"2024-09-10T20:30:33.862879+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:33.862879+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2610336,"num_objects":319,"num_object_clones":0,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11972,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11742","reported_seq":15724,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:35.160482+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:35.160482+0000","last_peered":"2024-09-10T20:30:35.160482+0000","last_clean":"2024-09-10T20:30:35.160482+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:35.160482+0000","last_undegraded":"2024-09-10T20:30:35.160482+0000","last_fullsized":"2024-09-10T20:30:35.160482+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2120272,"num_objects":259,"num_object_clones":0,"num_object_copies":1036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11867","reported_seq":15846,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:37.164982+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:37.164982+0000","last_peered":"2024-09-10T20:30:37.164982+0000","last_clean":"2024-09-10T20:30:37.164982+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:37.164982+0000","last_undegraded":"2024-09-10T20:30:37.164982+0000","last_fullsized":"2024-09-10T20:30:37.164982+0000","mapping_epoch":39,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11832","reported_seq":12115,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.366625+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:35.366625+0000","last_peered":"2024-09-10T20:30:35.366625+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:35.366625+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2297584,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11842","reported_seq":12150,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:35.402307+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:35.402307+0000","last_peered":"2024-09-10T20:30:35.402307+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:35.402307+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2644560,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11585","reported_seq":11777,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.201804+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:37.201804+0000","last_peered":"2024-09-10T20:30:37.201804+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:37.201804+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013776,"num_objects":246,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11789","reported_seq":15848,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.236655+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:34.236655+0000","last_peered":"2024-09-10T20:30:34.236655+0000","last_clean":"2024-09-10T20:30:34.236655+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:34.236655+0000","last_undegraded":"2024-09-10T20:30:34.236655+0000","last_fullsized":"2024-09-10T20:30:34.236655+0000","mapping_epoch":43,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2482176,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12170","reported_seq":12385,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.117065+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:37.117065+0000","last_peered":"2024-09-10T20:30:37.117065+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:37.117065+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2282656,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11798","reported_seq":15902,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:34.261242+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:34.261242+0000","last_peered":"2024-09-10T20:30:34.261242+0000","last_clean":"2024-09-10T20:30:34.261242+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:34.261242+0000","last_undegraded":"2024-09-10T20:30:34.261242+0000","last_fullsized":"2024-09-10T20:30:34.261242+0000","mapping_epoch":41,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2662400,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2671,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":38423240,"num_objects":4694,"num_object_clones":0,"num_object_copies":18776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149476,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1675,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77938688,"data_stored":77938688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161073,"ondisk_log_size":161073,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696632,"kb_used_data":19348,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93675208,"statfs":{"total":96636764160,"available":95923412992,"internally_reserved":0,"allocated":19812352,"data_stored":19580307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703000,"kb_used_data":17924,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93668840,"statfs":{"total":96636764160,"available":95916892160,"internally_reserved":0,"allocated":18354176,"data_stored":18122131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579644,"kb_used_data":20396,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93792196,"statfs":{"total":96636764160,"available":96043208704,"internally_reserved":0,"allocated":20885504,"data_stored":20653459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701956,"kb_used_data":20596,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93669884,"statfs":{"total":96636764160,"available":95917961216,"internally_reserved":0,"allocated":21090304,"data_stored":20858259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20586496,"data_stored":20586496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20340736,"data_stored":20340736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17829888,"data_stored":17829888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19181568,"data_stored":19181568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:42.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:42.677 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:42.702 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-10T20:30:42.134495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32254664,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150229,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681496,"kb_used_data":66032,"kb_used_omap":14,"kb_used_meta":2615345,"kb_avail":374805864,"statfs":{"total":386547056640,"available":383801204736,"internally_reserved":0,"allocated":67616768,"data_stored":66688588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2678113846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,18,12,20,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20094976,"num_objects":-2453,"num_object_clones":0,"num_object_copies":-9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004333"},"pg_stats":[{"pgid":"1.f","version":"44'11906","reported_seq":12196,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.404497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:40.404497+0000","last_peered":"2024-09-10T20:30:40.404497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:40.404497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12024","reported_seq":12164,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.395694+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:39.395694+0000","last_peered":"2024-09-10T20:30:39.395694+0000","last_clean":"2024-09-10T20:30:39.395694+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:39.395694+0000","last_undegraded":"2024-09-10T20:30:39.395694+0000","last_fullsized":"2024-09-10T20:30:39.395694+0000","mapping_epoch":35,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11883","reported_seq":12096,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.048741+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:37.048741+0000","last_peered":"2024-09-10T20:30:37.048741+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:37.048741+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2103912,"num_objects":258,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11882,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12165","reported_seq":16344,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.203960+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:39.203960+0000","last_peered":"2024-09-10T20:30:39.203960+0000","last_clean":"2024-09-10T20:30:39.203960+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:39.203960+0000","last_undegraded":"2024-09-10T20:30:39.203960+0000","last_fullsized":"2024-09-10T20:30:39.203960+0000","mapping_epoch":37,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1974272,"num_objects":241,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6425,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11907","reported_seq":12237,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.359597+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:40.359597+0000","last_peered":"2024-09-10T20:30:40.359597+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:40.359597+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143392,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11846","reported_seq":12075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.490235+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:40.490235+0000","last_peered":"2024-09-10T20:30:40.490235+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:40.490235+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2064384,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11846,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11816","reported_seq":13876,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:38.945733+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:38.945733+0000","last_peered":"2024-09-10T20:30:38.945733+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:38.945733+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2077856,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11814,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-718}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12024","reported_seq":12202,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:38.895887+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:38.895887+0000","last_peered":"2024-09-10T20:30:38.895887+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:38.895887+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2200736,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11800","reported_seq":15782,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:40.466992+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:40.466992+0000","last_peered":"2024-09-10T20:30:40.466992+0000","last_clean":"2024-09-10T20:30:40.466992+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:40.466992+0000","last_undegraded":"2024-09-10T20:30:40.466992+0000","last_fullsized":"2024-09-10T20:30:40.466992+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1645136,"num_objects":201,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11867","reported_seq":15846,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:37.164982+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:37.164982+0000","last_peered":"2024-09-10T20:30:37.164982+0000","last_clean":"2024-09-10T20:30:37.164982+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:37.164982+0000","last_undegraded":"2024-09-10T20:30:37.164982+0000","last_fullsized":"2024-09-10T20:30:37.164982+0000","mapping_epoch":39,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11894","reported_seq":12177,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.428006+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:40.428006+0000","last_peered":"2024-09-10T20:30:40.428006+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:40.428006+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1789680,"num_objects":219,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11917","reported_seq":12223,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.427321+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:40.427321+0000","last_peered":"2024-09-10T20:30:40.427321+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:40.427321+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2030160,"num_objects":248,"num_object_clones":0,"num_object_copies":992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11916,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11585","reported_seq":11777,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.201804+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:37.201804+0000","last_peered":"2024-09-10T20:30:37.201804+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:37.201804+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013776,"num_objects":246,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11853","reported_seq":15913,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.302486+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:39.302486+0000","last_peered":"2024-09-10T20:30:39.302486+0000","last_clean":"2024-09-10T20:30:39.302486+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:39.302486+0000","last_undegraded":"2024-09-10T20:30:39.302486+0000","last_fullsized":"2024-09-10T20:30:39.302486+0000","mapping_epoch":43,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1957888,"num_objects":239,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12170","reported_seq":12385,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.117065+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:37.117065+0000","last_peered":"2024-09-10T20:30:37.117065+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:37.117065+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2282656,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11869","reported_seq":15974,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.283316+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:39.283316+0000","last_peered":"2024-09-10T20:30:39.283316+0000","last_clean":"2024-09-10T20:30:39.283316+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:39.283316+0000","last_undegraded":"2024-09-10T20:30:39.283316+0000","last_fullsized":"2024-09-10T20:30:39.283316+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2080768,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":32254664,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150229,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65417216,"data_stored":65417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696812,"kb_used_data":15384,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93675028,"statfs":{"total":96636764160,"available":95923228672,"internally_reserved":0,"allocated":15753216,"data_stored":15521171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,4,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703000,"kb_used_data":17924,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93668840,"statfs":{"total":96636764160,"available":95916892160,"internally_reserved":0,"allocated":18354176,"data_stored":18122131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579720,"kb_used_data":16292,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93792120,"statfs":{"total":96636764160,"available":96043130880,"internally_reserved":0,"allocated":16683008,"data_stored":16450963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576909010},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215244,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701964,"kb_used_data":16432,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93669876,"statfs":{"total":96636764160,"available":95917953024,"internally_reserved":0,"allocated":16826368,"data_stored":16594323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16396288,"data_stored":16396288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16076800,"data_stored":16076800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17829888,"data_stored":17829888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15114240,"data_stored":15114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:42.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:43.372 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:43.373 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:43.401 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-10T20:30:42.134495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32254664,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150229,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681496,"kb_used_data":66032,"kb_used_omap":14,"kb_used_meta":2615345,"kb_avail":374805864,"statfs":{"total":386547056640,"available":383801204736,"internally_reserved":0,"allocated":67616768,"data_stored":66688588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2678113846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,18,12,20,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20094976,"num_objects":-2453,"num_object_clones":0,"num_object_copies":-9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004333"},"pg_stats":[{"pgid":"1.f","version":"44'11906","reported_seq":12196,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.404497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:40.404497+0000","last_peered":"2024-09-10T20:30:40.404497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:40.404497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12024","reported_seq":12164,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.395694+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:39.395694+0000","last_peered":"2024-09-10T20:30:39.395694+0000","last_clean":"2024-09-10T20:30:39.395694+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:39.395694+0000","last_undegraded":"2024-09-10T20:30:39.395694+0000","last_fullsized":"2024-09-10T20:30:39.395694+0000","mapping_epoch":35,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11883","reported_seq":12096,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.048741+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:37.048741+0000","last_peered":"2024-09-10T20:30:37.048741+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:37.048741+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2103912,"num_objects":258,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11882,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12165","reported_seq":16344,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.203960+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:39.203960+0000","last_peered":"2024-09-10T20:30:39.203960+0000","last_clean":"2024-09-10T20:30:39.203960+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:39.203960+0000","last_undegraded":"2024-09-10T20:30:39.203960+0000","last_fullsized":"2024-09-10T20:30:39.203960+0000","mapping_epoch":37,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1974272,"num_objects":241,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6425,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11907","reported_seq":12237,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.359597+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:40.359597+0000","last_peered":"2024-09-10T20:30:40.359597+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:40.359597+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143392,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11846","reported_seq":12075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.490235+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:40.490235+0000","last_peered":"2024-09-10T20:30:40.490235+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:40.490235+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2064384,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11846,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11816","reported_seq":13876,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:38.945733+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:38.945733+0000","last_peered":"2024-09-10T20:30:38.945733+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:38.945733+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2077856,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11814,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-718}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12024","reported_seq":12202,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:38.895887+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:38.895887+0000","last_peered":"2024-09-10T20:30:38.895887+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:38.895887+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2200736,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11800","reported_seq":15782,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:40.466992+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:40.466992+0000","last_peered":"2024-09-10T20:30:40.466992+0000","last_clean":"2024-09-10T20:30:40.466992+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:40.466992+0000","last_undegraded":"2024-09-10T20:30:40.466992+0000","last_fullsized":"2024-09-10T20:30:40.466992+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1645136,"num_objects":201,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11867","reported_seq":15846,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:37.164982+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:37.164982+0000","last_peered":"2024-09-10T20:30:37.164982+0000","last_clean":"2024-09-10T20:30:37.164982+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:37.164982+0000","last_undegraded":"2024-09-10T20:30:37.164982+0000","last_fullsized":"2024-09-10T20:30:37.164982+0000","mapping_epoch":39,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11894","reported_seq":12177,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.428006+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:40.428006+0000","last_peered":"2024-09-10T20:30:40.428006+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:40.428006+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1789680,"num_objects":219,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11917","reported_seq":12223,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.427321+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:40.427321+0000","last_peered":"2024-09-10T20:30:40.427321+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:40.427321+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2030160,"num_objects":248,"num_object_clones":0,"num_object_copies":992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11916,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11585","reported_seq":11777,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.201804+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:37.201804+0000","last_peered":"2024-09-10T20:30:37.201804+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:37.201804+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013776,"num_objects":246,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11853","reported_seq":15913,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.302486+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:39.302486+0000","last_peered":"2024-09-10T20:30:39.302486+0000","last_clean":"2024-09-10T20:30:39.302486+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:39.302486+0000","last_undegraded":"2024-09-10T20:30:39.302486+0000","last_fullsized":"2024-09-10T20:30:39.302486+0000","mapping_epoch":43,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1957888,"num_objects":239,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12170","reported_seq":12385,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:37.117065+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:37.117065+0000","last_peered":"2024-09-10T20:30:37.117065+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:37.117065+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2282656,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11869","reported_seq":15974,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.283316+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:39.283316+0000","last_peered":"2024-09-10T20:30:39.283316+0000","last_clean":"2024-09-10T20:30:39.283316+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:39.283316+0000","last_undegraded":"2024-09-10T20:30:39.283316+0000","last_fullsized":"2024-09-10T20:30:39.283316+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2080768,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":32254664,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150229,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65417216,"data_stored":65417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696812,"kb_used_data":15384,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93675028,"statfs":{"total":96636764160,"available":95923228672,"internally_reserved":0,"allocated":15753216,"data_stored":15521171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,4,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703000,"kb_used_data":17924,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93668840,"statfs":{"total":96636764160,"available":95916892160,"internally_reserved":0,"allocated":18354176,"data_stored":18122131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579720,"kb_used_data":16292,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93792120,"statfs":{"total":96636764160,"available":96043130880,"internally_reserved":0,"allocated":16683008,"data_stored":16450963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576909010},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215244,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701964,"kb_used_data":16432,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93669876,"statfs":{"total":96636764160,"available":95917953024,"internally_reserved":0,"allocated":16826368,"data_stored":16594323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16396288,"data_stored":16396288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16076800,"data_stored":16076800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17829888,"data_stored":17829888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15114240,"data_stored":15114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:43.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11906", 'reported_seq': 12196, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:40.404497+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:40.404497+0000', 'last_peered': '2024-09-10T20:30:40.404497+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:40.404497+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1769472, 'num_objects': 216, 'num_object_clones': 0, 'num_object_copies': 864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11906, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1639}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'11883", 'reported_seq': 12096, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:37.048741+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:37.048741+0000', 'last_peered': '2024-09-10T20:30:37.048741+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:37.048741+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2103912, 'num_objects': 258, 'num_object_clones': 0, 'num_object_copies': 1032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 258, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11882, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1650}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11907", 'reported_seq': 12237, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:40.359597+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:40.359597+0000', 'last_peered': '2024-09-10T20:30:40.359597+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:40.359597+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2143392, 'num_objects': 262, 'num_object_clones': 0, 'num_object_copies': 1048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11905, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1593}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11846", 'reported_seq': 12075, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:40.490235+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:40.490235+0000', 'last_peered': '2024-09-10T20:30:40.490235+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:40.490235+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2064384, 'num_objects': 252, 'num_object_clones': 0, 'num_object_copies': 1008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11846, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1605}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:43.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11816", 'reported_seq': 13876, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:38.945733+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:38.945733+0000', 'last_peered': '2024-09-10T20:30:38.945733+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:38.945733+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2077856, 'num_objects': 254, 'num_object_clones': 0, 'num_object_copies': 1016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 972, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11814, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 985, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -718}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:43.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:43.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12024", 'reported_seq': 12202, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:38.895887+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:38.895887+0000', 'last_peered': '2024-09-10T20:30:38.895887+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:38.895887+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2200736, 'num_objects': 269, 'num_object_clones': 0, 'num_object_copies': 1076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12022, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:43.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:43.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11894", 'reported_seq': 12177, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:40.428006+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:40.428006+0000', 'last_peered': '2024-09-10T20:30:40.428006+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:40.428006+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1789680, 'num_objects': 219, 'num_object_clones': 0, 'num_object_copies': 876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11891, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1637}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:43.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:43.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11917", 'reported_seq': 12223, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:40.427321+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:40.427321+0000', 'last_peered': '2024-09-10T20:30:40.427321+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:40.427321+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2030160, 'num_objects': 248, 'num_object_clones': 0, 'num_object_copies': 992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11916, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:43.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:43.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11585", 'reported_seq': 11777, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:37.201804+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:37.201804+0000', 'last_peered': '2024-09-10T20:30:37.201804+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:37.201804+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2013776, 'num_objects': 246, 'num_object_clones': 0, 'num_object_copies': 984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11584, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:43.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:43.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12170", 'reported_seq': 12385, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:37.117065+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:37.117065+0000', 'last_peered': '2024-09-10T20:30:37.117065+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:37.117065+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2282656, 'num_objects': 279, 'num_object_clones': 0, 'num_object_copies': 1116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12168, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:43.405 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:44.153 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:44.153 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17986/15764 objects degraded (114.095%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":732,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":3941,"data_bytes":32254664,"bytes_used":2745851904,"bytes_avail":383801204736,"bytes_total":386547056640,"degraded_objects":17986,"degraded_total":15764,"degraded_ratio":1.1409540725704137,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:44.175 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:44.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:44.772 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:44.797 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-10T20:30:44.135161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29149896,"num_objects":3562,"num_object_clones":0,"num_object_copies":14248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150608,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717700,"kb_used_data":57836,"kb_used_omap":14,"kb_used_meta":2659761,"kb_avail":374769660,"statfs":{"total":386547056640,"available":383764131840,"internally_reserved":0,"allocated":59224064,"data_stored":58295884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2723595830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,18,21,18,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17793024,"num_objects":-2172,"num_object_clones":0,"num_object_copies":-8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004172"},"pg_stats":[{"pgid":"1.f","version":"44'11906","reported_seq":12196,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.404497+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:40.404497+0000","last_peered":"2024-09-10T20:30:40.404497+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:40.404497+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12024","reported_seq":12164,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.395694+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:39.395694+0000","last_peered":"2024-09-10T20:30:39.395694+0000","last_clean":"2024-09-10T20:30:39.395694+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:39.395694+0000","last_undegraded":"2024-09-10T20:30:39.395694+0000","last_fullsized":"2024-09-10T20:30:39.395694+0000","mapping_epoch":35,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'11941","reported_seq":12153,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:42.097033+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:42.097033+0000","last_peered":"2024-09-10T20:30:42.097033+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:42.097033+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1628776,"num_objects":200,"num_object_clones":0,"num_object_copies":800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11940,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1708}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12165","reported_seq":16344,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.203960+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:39.203960+0000","last_peered":"2024-09-10T20:30:39.203960+0000","last_clean":"2024-09-10T20:30:39.203960+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:39.203960+0000","last_undegraded":"2024-09-10T20:30:39.203960+0000","last_fullsized":"2024-09-10T20:30:39.203960+0000","mapping_epoch":37,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1974272,"num_objects":241,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6425,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11907","reported_seq":12237,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.359597+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:40.359597+0000","last_peered":"2024-09-10T20:30:40.359597+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:40.359597+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143392,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11846","reported_seq":12075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.490235+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:40.490235+0000","last_peered":"2024-09-10T20:30:40.490235+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:40.490235+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2064384,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11846,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11891","reported_seq":14095,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:44.077351+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:44.077351+0000","last_peered":"2024-09-10T20:30:44.077351+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:44.077351+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1463456,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12088","reported_seq":12266,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:43.943632+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:43.943632+0000","last_peered":"2024-09-10T20:30:43.943632+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:43.943632+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1676448,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11800","reported_seq":15782,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:40.466992+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:40.466992+0000","last_peered":"2024-09-10T20:30:40.466992+0000","last_clean":"2024-09-10T20:30:40.466992+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:40.466992+0000","last_undegraded":"2024-09-10T20:30:40.466992+0000","last_fullsized":"2024-09-10T20:30:40.466992+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1645136,"num_objects":201,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'11929","reported_seq":15909,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:42.152399+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:42.152399+0000","last_peered":"2024-09-10T20:30:42.152399+0000","last_clean":"2024-09-10T20:30:42.152399+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:42.152399+0000","last_undegraded":"2024-09-10T20:30:42.152399+0000","last_fullsized":"2024-09-10T20:30:42.152399+0000","mapping_epoch":39,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1744896,"num_objects":213,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11894","reported_seq":12177,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.428006+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:40.428006+0000","last_peered":"2024-09-10T20:30:40.428006+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:40.428006+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1789680,"num_objects":219,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11917","reported_seq":12223,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:40.427321+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:40.427321+0000","last_peered":"2024-09-10T20:30:40.427321+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:40.427321+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2030160,"num_objects":248,"num_object_clones":0,"num_object_copies":992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11916,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11645","reported_seq":11838,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:41.967318+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:41.967318+0000","last_peered":"2024-09-10T20:30:41.967318+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:41.967318+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1522256,"num_objects":186,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11644,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1717}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11853","reported_seq":15913,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.302486+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:39.302486+0000","last_peered":"2024-09-10T20:30:39.302486+0000","last_clean":"2024-09-10T20:30:39.302486+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:39.302486+0000","last_undegraded":"2024-09-10T20:30:39.302486+0000","last_fullsized":"2024-09-10T20:30:39.302486+0000","mapping_epoch":43,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1957888,"num_objects":239,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12230","reported_seq":12445,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:42.059746+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:42.059746+0000","last_peered":"2024-09-10T20:30:42.059746+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:42.059746+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1791136,"num_objects":219,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1689}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11869","reported_seq":15974,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:39.283316+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:39.283316+0000","last_peered":"2024-09-10T20:30:39.283316+0000","last_clean":"2024-09-10T20:30:39.283316+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:39.283316+0000","last_undegraded":"2024-09-10T20:30:39.283316+0000","last_fullsized":"2024-09-10T20:30:39.283316+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2080768,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":29149896,"num_objects":3562,"num_object_clones":0,"num_object_copies":14248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150608,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57008128,"data_stored":57008128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696812,"kb_used_data":15384,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93675028,"statfs":{"total":96636764160,"available":95923228672,"internally_reserved":0,"allocated":15753216,"data_stored":15521171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,4,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721148,"kb_used_data":13868,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93650692,"statfs":{"total":96636764160,"available":95898308608,"internally_reserved":0,"allocated":14200832,"data_stored":13968787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579720,"kb_used_data":16292,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93792120,"statfs":{"total":96636764160,"available":96043130880,"internally_reserved":0,"allocated":16683008,"data_stored":16450963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576909010},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720020,"kb_used_data":12292,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93651820,"statfs":{"total":96636764160,"available":95899463680,"internally_reserved":0,"allocated":12587008,"data_stored":12354963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724692687},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12152832,"data_stored":12152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16076800,"data_stored":16076800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13664256,"data_stored":13664256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15114240,"data_stored":15114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:47.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:48.430 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:48.448 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-10T20:30:48.136602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21883592,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151495,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160992,"ondisk_log_size":160992,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737132,"kb_used_data":45348,"kb_used_omap":14,"kb_used_meta":2691697,"kb_avail":374750228,"statfs":{"total":386547056640,"available":383744233472,"internally_reserved":0,"allocated":46436352,"data_stored":45508172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2756298294},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,6,15,10,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18685952,"num_objects":-2281,"num_object_clones":0,"num_object_copies":-9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004287"},"pg_stats":[{"pgid":"1.f","version":"44'11964","reported_seq":12255,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.309508+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:45.309508+0000","last_peered":"2024-09-10T20:30:45.309508+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:45.309508+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1294336,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12087","reported_seq":12229,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.312611+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:44.312611+0000","last_peered":"2024-09-10T20:30:44.312611+0000","last_clean":"2024-09-10T20:30:44.312611+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:44.312611+0000","last_undegraded":"2024-09-10T20:30:44.312611+0000","last_fullsized":"2024-09-10T20:30:44.312611+0000","mapping_epoch":35,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1351680,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12006","reported_seq":12219,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:47.048143+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:47.048143+0000","last_peered":"2024-09-10T20:30:47.048143+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:47.048143+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1096296,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12005,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12224","reported_seq":16403,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.106132+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:44.106132+0000","last_peered":"2024-09-10T20:30:44.106132+0000","last_clean":"2024-09-10T20:30:44.106132+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:44.106132+0000","last_undegraded":"2024-09-10T20:30:44.106132+0000","last_fullsized":"2024-09-10T20:30:44.106132+0000","mapping_epoch":37,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1490944,"num_objects":182,"num_object_clones":0,"num_object_copies":728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11971","reported_seq":12301,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.501062+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:45.501062+0000","last_peered":"2024-09-10T20:30:45.501062+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:45.501062+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1619104,"num_objects":198,"num_object_clones":0,"num_object_copies":792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11918","reported_seq":12149,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.492908+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:45.492908+0000","last_peered":"2024-09-10T20:30:45.492908+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:45.492908+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1474560,"num_objects":180,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11891","reported_seq":14095,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:44.077351+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:44.077351+0000","last_peered":"2024-09-10T20:30:44.077351+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:44.077351+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1463456,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12088","reported_seq":12266,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:43.943632+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:43.943632+0000","last_peered":"2024-09-10T20:30:43.943632+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:43.943632+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1676448,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11847","reported_seq":15828,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:45.565333+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:45.565333+0000","last_peered":"2024-09-10T20:30:45.565333+0000","last_clean":"2024-09-10T20:30:45.565333+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:45.565333+0000","last_undegraded":"2024-09-10T20:30:45.565333+0000","last_fullsized":"2024-09-10T20:30:45.565333+0000","mapping_epoch":33,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1260112,"num_objects":154,"num_object_clones":0,"num_object_copies":616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12000","reported_seq":15978,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:47.135910+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:47.135910+0000","last_peered":"2024-09-10T20:30:47.135910+0000","last_clean":"2024-09-10T20:30:47.135910+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:47.135910+0000","last_undegraded":"2024-09-10T20:30:47.135910+0000","last_fullsized":"2024-09-10T20:30:47.135910+0000","mapping_epoch":39,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163264,"num_objects":142,"num_object_clones":0,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11952","reported_seq":12235,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.455748+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:45.455748+0000","last_peered":"2024-09-10T20:30:45.455748+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:45.455748+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1314544,"num_objects":161,"num_object_clones":0,"num_object_copies":644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11993","reported_seq":12301,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.412927+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:45.412927+0000","last_peered":"2024-09-10T20:30:45.412927+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:45.412927+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1407568,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1688}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11698","reported_seq":11891,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:47.066722+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:47.066722+0000","last_peered":"2024-09-10T20:30:47.066722+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:47.066722+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088080,"num_objects":133,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11697,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1770}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11930","reported_seq":15990,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.466794+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:44.466794+0000","last_peered":"2024-09-10T20:30:44.466794+0000","last_clean":"2024-09-10T20:30:44.466794+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:44.466794+0000","last_undegraded":"2024-09-10T20:30:44.466794+0000","last_fullsized":"2024-09-10T20:30:44.466794+0000","mapping_epoch":43,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1327104,"num_objects":162,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12293","reported_seq":12508,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:46.971244+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:46.971244+0000","last_peered":"2024-09-10T20:30:46.971244+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:46.971244+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1275040,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12291,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11930","reported_seq":16035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.203484+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:44.203484+0000","last_peered":"2024-09-10T20:30:44.203484+0000","last_clean":"2024-09-10T20:30:44.203484+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:44.203484+0000","last_undegraded":"2024-09-10T20:30:44.203484+0000","last_fullsized":"2024-09-10T20:30:44.203484+0000","mapping_epoch":41,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1581056,"num_objects":193,"num_object_clones":0,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":21883592,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151495,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44249088,"data_stored":44249088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160992,"ondisk_log_size":160992,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714760,"kb_used_data":11124,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93657080,"statfs":{"total":96636764160,"available":95904849920,"internally_reserved":0,"allocated":11390976,"data_stored":11158931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721160,"kb_used_data":9716,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":93650680,"statfs":{"total":96636764160,"available":95898296320,"internally_reserved":0,"allocated":9949184,"data_stored":9717139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581192,"kb_used_data":12216,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93790648,"statfs":{"total":96636764160,"available":96041623552,"internally_reserved":0,"allocated":12509184,"data_stored":12277139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582610642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720020,"kb_used_data":12292,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93651820,"statfs":{"total":96636764160,"available":95899463680,"internally_reserved":0,"allocated":12587008,"data_stored":12354963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724692687},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12152832,"data_stored":12152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11866112,"data_stored":11866112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9396224,"data_stored":9396224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10833920,"data_stored":10833920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:48.449 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:49.093 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:49.118 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-10T20:30:48.136602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21883592,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151495,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160992,"ondisk_log_size":160992,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737132,"kb_used_data":45348,"kb_used_omap":14,"kb_used_meta":2691697,"kb_avail":374750228,"statfs":{"total":386547056640,"available":383744233472,"internally_reserved":0,"allocated":46436352,"data_stored":45508172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2756298294},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,6,15,10,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18685952,"num_objects":-2281,"num_object_clones":0,"num_object_copies":-9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004287"},"pg_stats":[{"pgid":"1.f","version":"44'11964","reported_seq":12255,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.309508+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:45.309508+0000","last_peered":"2024-09-10T20:30:45.309508+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:45.309508+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1294336,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12087","reported_seq":12229,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.312611+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:44.312611+0000","last_peered":"2024-09-10T20:30:44.312611+0000","last_clean":"2024-09-10T20:30:44.312611+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:44.312611+0000","last_undegraded":"2024-09-10T20:30:44.312611+0000","last_fullsized":"2024-09-10T20:30:44.312611+0000","mapping_epoch":35,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1351680,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12006","reported_seq":12219,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:47.048143+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:47.048143+0000","last_peered":"2024-09-10T20:30:47.048143+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:47.048143+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1096296,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12005,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12224","reported_seq":16403,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.106132+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:44.106132+0000","last_peered":"2024-09-10T20:30:44.106132+0000","last_clean":"2024-09-10T20:30:44.106132+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:44.106132+0000","last_undegraded":"2024-09-10T20:30:44.106132+0000","last_fullsized":"2024-09-10T20:30:44.106132+0000","mapping_epoch":37,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1490944,"num_objects":182,"num_object_clones":0,"num_object_copies":728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11971","reported_seq":12301,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.501062+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:45.501062+0000","last_peered":"2024-09-10T20:30:45.501062+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:45.501062+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1619104,"num_objects":198,"num_object_clones":0,"num_object_copies":792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11918","reported_seq":12149,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.492908+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:45.492908+0000","last_peered":"2024-09-10T20:30:45.492908+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:45.492908+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1474560,"num_objects":180,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11891","reported_seq":14095,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:44.077351+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:44.077351+0000","last_peered":"2024-09-10T20:30:44.077351+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:44.077351+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1463456,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12088","reported_seq":12266,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:43.943632+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:43.943632+0000","last_peered":"2024-09-10T20:30:43.943632+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:43.943632+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"29'2000","ondisk_log_start":"29'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1676448,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11847","reported_seq":15828,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:45.565333+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:45.565333+0000","last_peered":"2024-09-10T20:30:45.565333+0000","last_clean":"2024-09-10T20:30:45.565333+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:45.565333+0000","last_undegraded":"2024-09-10T20:30:45.565333+0000","last_fullsized":"2024-09-10T20:30:45.565333+0000","mapping_epoch":33,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1260112,"num_objects":154,"num_object_clones":0,"num_object_copies":616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12000","reported_seq":15978,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:47.135910+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:47.135910+0000","last_peered":"2024-09-10T20:30:47.135910+0000","last_clean":"2024-09-10T20:30:47.135910+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:47.135910+0000","last_undegraded":"2024-09-10T20:30:47.135910+0000","last_fullsized":"2024-09-10T20:30:47.135910+0000","mapping_epoch":39,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163264,"num_objects":142,"num_object_clones":0,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11952","reported_seq":12235,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.455748+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:45.455748+0000","last_peered":"2024-09-10T20:30:45.455748+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:45.455748+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1314544,"num_objects":161,"num_object_clones":0,"num_object_copies":644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11993","reported_seq":12301,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.412927+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:45.412927+0000","last_peered":"2024-09-10T20:30:45.412927+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:45.412927+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1407568,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1688}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11698","reported_seq":11891,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:47.066722+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:47.066722+0000","last_peered":"2024-09-10T20:30:47.066722+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:47.066722+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088080,"num_objects":133,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11697,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1770}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11930","reported_seq":15990,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.466794+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:44.466794+0000","last_peered":"2024-09-10T20:30:44.466794+0000","last_clean":"2024-09-10T20:30:44.466794+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:44.466794+0000","last_undegraded":"2024-09-10T20:30:44.466794+0000","last_fullsized":"2024-09-10T20:30:44.466794+0000","mapping_epoch":43,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1327104,"num_objects":162,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12293","reported_seq":12508,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:46.971244+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:46.971244+0000","last_peered":"2024-09-10T20:30:46.971244+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:46.971244+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1275040,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12291,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11930","reported_seq":16035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:44.203484+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:44.203484+0000","last_peered":"2024-09-10T20:30:44.203484+0000","last_clean":"2024-09-10T20:30:44.203484+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:44.203484+0000","last_undegraded":"2024-09-10T20:30:44.203484+0000","last_fullsized":"2024-09-10T20:30:44.203484+0000","mapping_epoch":41,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1581056,"num_objects":193,"num_object_clones":0,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":21883592,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151495,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44249088,"data_stored":44249088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160992,"ondisk_log_size":160992,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714760,"kb_used_data":11124,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93657080,"statfs":{"total":96636764160,"available":95904849920,"internally_reserved":0,"allocated":11390976,"data_stored":11158931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721160,"kb_used_data":9716,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":93650680,"statfs":{"total":96636764160,"available":95898296320,"internally_reserved":0,"allocated":9949184,"data_stored":9717139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581192,"kb_used_data":12216,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93790648,"statfs":{"total":96636764160,"available":96041623552,"internally_reserved":0,"allocated":12509184,"data_stored":12277139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582610642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720020,"kb_used_data":12292,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93651820,"statfs":{"total":96636764160,"available":95899463680,"internally_reserved":0,"allocated":12587008,"data_stored":12354963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724692687},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12152832,"data_stored":12152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11866112,"data_stored":11866112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9396224,"data_stored":9396224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10833920,"data_stored":10833920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:49.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:49.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'11964", 'reported_seq': 12255, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:45.309508+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:45.309508+0000', 'last_peered': '2024-09-10T20:30:45.309508+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:45.309508+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1294336, 'num_objects': 158, 'num_object_clones': 0, 'num_object_copies': 632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1697}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12006", 'reported_seq': 12219, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:47.048143+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:47.048143+0000', 'last_peered': '2024-09-10T20:30:47.048143+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:47.048143+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1096296, 'num_objects': 135, 'num_object_clones': 0, 'num_object_copies': 540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 135, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12005, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1773}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'11971", 'reported_seq': 12301, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:45.501062+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:45.501062+0000', 'last_peered': '2024-09-10T20:30:45.501062+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:45.501062+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1619104, 'num_objects': 198, 'num_object_clones': 0, 'num_object_copies': 792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11969, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1657}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11918", 'reported_seq': 12149, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:45.492908+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:45.492908+0000', 'last_peered': '2024-09-10T20:30:45.492908+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:45.492908+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1474560, 'num_objects': 180, 'num_object_clones': 0, 'num_object_copies': 720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11918, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1677}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11891", 'reported_seq': 14095, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:44.077351+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:44.077351+0000', 'last_peered': '2024-09-10T20:30:44.077351+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:44.077351+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1463456, 'num_objects': 179, 'num_object_clones': 0, 'num_object_copies': 716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 900, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11889, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1057, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -721}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12088", 'reported_seq': 12266, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:43.943632+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:43.943632+0000', 'last_peered': '2024-09-10T20:30:43.943632+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:43.943632+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1676448, 'num_objects': 205, 'num_object_clones': 0, 'num_object_copies': 820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12086, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:49.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'11952", 'reported_seq': 12235, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:45.455748+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:45.455748+0000', 'last_peered': '2024-09-10T20:30:45.455748+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:45.455748+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1314544, 'num_objects': 161, 'num_object_clones': 0, 'num_object_copies': 644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11949, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1695}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:49.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:49.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'11993", 'reported_seq': 12301, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:45.412927+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:45.412927+0000', 'last_peered': '2024-09-10T20:30:45.412927+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:45.412927+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1407568, 'num_objects': 172, 'num_object_clones': 0, 'num_object_copies': 688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11992, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1688}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:49.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:49.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11698", 'reported_seq': 11891, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:47.066722+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:47.066722+0000', 'last_peered': '2024-09-10T20:30:47.066722+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:47.066722+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1088080, 'num_objects': 133, 'num_object_clones': 0, 'num_object_copies': 532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11697, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1770}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:49.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:49.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12293", 'reported_seq': 12508, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:46.971244+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:46.971244+0000', 'last_peered': '2024-09-10T20:30:46.971244+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:46.971244+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2200", 'ondisk_log_start': "32'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1275040, 'num_objects': 156, 'num_object_clones': 0, 'num_object_copies': 624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12291, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:49.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:49.873 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17914/10700 objects degraded (167.421%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":738,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2675,"data_bytes":21883592,"bytes_used":2802823168,"bytes_avail":383744233472,"bytes_total":386547056640,"degraded_objects":17914,"degraded_total":10700,"degraded_ratio":1.6742056074766356,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":190},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:49.893 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:49.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:50.500 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:50.522 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-10T20:30:50.137423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":18614984,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151894,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161091,"ondisk_log_size":161091,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737108,"kb_used_data":36980,"kb_used_omap":14,"kb_used_meta":2700017,"kb_avail":374750252,"statfs":{"total":386547056640,"available":383744258048,"internally_reserved":0,"allocated":37867520,"data_stored":36939340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2764817974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,9,9,9,19,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19808256,"num_objects":-2418,"num_object_clones":0,"num_object_copies":-9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004359"},"pg_stats":[{"pgid":"1.f","version":"44'11964","reported_seq":12255,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.309508+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:45.309508+0000","last_peered":"2024-09-10T20:30:45.309508+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:45.309508+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1294336,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12153","reported_seq":12295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.225533+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:49.225533+0000","last_peered":"2024-09-10T20:30:49.225533+0000","last_clean":"2024-09-10T20:30:49.225533+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:49.225533+0000","last_undegraded":"2024-09-10T20:30:49.225533+0000","last_fullsized":"2024-09-10T20:30:49.225533+0000","mapping_epoch":35,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":811008,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8594,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12006","reported_seq":12219,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:47.048143+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:47.048143+0000","last_peered":"2024-09-10T20:30:47.048143+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:47.048143+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1096296,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12005,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12296","reported_seq":16475,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.474556+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:49.474556+0000","last_peered":"2024-09-10T20:30:49.474556+0000","last_clean":"2024-09-10T20:30:49.474556+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:49.474556+0000","last_undegraded":"2024-09-10T20:30:49.474556+0000","last_fullsized":"2024-09-10T20:30:49.474556+0000","mapping_epoch":37,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":901120,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'11971","reported_seq":12301,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.501062+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:45.501062+0000","last_peered":"2024-09-10T20:30:45.501062+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:45.501062+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1619104,"num_objects":198,"num_object_clones":0,"num_object_copies":792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11918","reported_seq":12149,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.492908+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:45.492908+0000","last_peered":"2024-09-10T20:30:45.492908+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:45.492908+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1474560,"num_objects":180,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11951","reported_seq":14305,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:49.147968+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:49.147968+0000","last_peered":"2024-09-10T20:30:49.147968+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:49.147968+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":971936,"num_objects":119,"num_object_clones":0,"num_object_copies":476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-706}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12161","reported_seq":12339,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:48.972623+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:48.972623+0000","last_peered":"2024-09-10T20:30:48.972623+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:48.972623+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1078432,"num_objects":132,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11847","reported_seq":15828,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:45.565333+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:45.565333+0000","last_peered":"2024-09-10T20:30:45.565333+0000","last_clean":"2024-09-10T20:30:45.565333+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:45.565333+0000","last_undegraded":"2024-09-10T20:30:45.565333+0000","last_fullsized":"2024-09-10T20:30:45.565333+0000","mapping_epoch":33,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1260112,"num_objects":154,"num_object_clones":0,"num_object_copies":616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12000","reported_seq":15978,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:47.135910+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:47.135910+0000","last_peered":"2024-09-10T20:30:47.135910+0000","last_clean":"2024-09-10T20:30:47.135910+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:47.135910+0000","last_undegraded":"2024-09-10T20:30:47.135910+0000","last_fullsized":"2024-09-10T20:30:47.135910+0000","mapping_epoch":39,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163264,"num_objects":142,"num_object_clones":0,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'11952","reported_seq":12235,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.455748+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:45.455748+0000","last_peered":"2024-09-10T20:30:45.455748+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:45.455748+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1314544,"num_objects":161,"num_object_clones":0,"num_object_copies":644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'11993","reported_seq":12301,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:45.412927+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:45.412927+0000","last_peered":"2024-09-10T20:30:45.412927+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:45.412927+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1407568,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1688}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11698","reported_seq":11891,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:47.066722+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:47.066722+0000","last_peered":"2024-09-10T20:30:47.066722+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:47.066722+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088080,"num_objects":133,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11697,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1770}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11990","reported_seq":16050,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.505435+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:49.505435+0000","last_peered":"2024-09-10T20:30:49.505435+0000","last_clean":"2024-09-10T20:30:49.505435+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:49.505435+0000","last_undegraded":"2024-09-10T20:30:49.505435+0000","last_fullsized":"2024-09-10T20:30:49.505435+0000","mapping_epoch":43,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":835584,"num_objects":102,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":942,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12293","reported_seq":12508,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:46.971244+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:46.971244+0000","last_peered":"2024-09-10T20:30:46.971244+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:46.971244+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1275040,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12291,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11998","reported_seq":16103,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.375987+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:49.375987+0000","last_peered":"2024-09-10T20:30:49.375987+0000","last_clean":"2024-09-10T20:30:49.375987+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:49.375987+0000","last_undegraded":"2024-09-10T20:30:49.375987+0000","last_fullsized":"2024-09-10T20:30:49.375987+0000","mapping_epoch":41,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2871,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":18614984,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151894,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35721216,"data_stored":35721216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161091,"ondisk_log_size":161091,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714760,"kb_used_data":11124,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93657080,"statfs":{"total":96636764160,"available":95904849920,"internally_reserved":0,"allocated":11390976,"data_stored":11158931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721160,"kb_used_data":9716,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":93650680,"statfs":{"total":96636764160,"available":95898296320,"internally_reserved":0,"allocated":9949184,"data_stored":9717139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581188,"kb_used_data":8052,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93790652,"statfs":{"total":96636764160,"available":96041627648,"internally_reserved":0,"allocated":8245248,"data_stored":8013203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720000,"kb_used_data":8088,"kb_used_omap":4,"kb_used_meta":711867,"kb_avail":93651840,"statfs":{"total":96636764160,"available":95899484160,"internally_reserved":0,"allocated":8282112,"data_stored":8050067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":728952527},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7868416,"data_stored":7868416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7622656,"data_stored":7622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9396224,"data_stored":9396224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10833920,"data_stored":10833920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:53.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:54.172 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:54.191 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-10T20:30:54.138873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13413064,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152529,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160926,"ondisk_log_size":160926,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2720848,"kb_used_data":29032,"kb_used_omap":14,"kb_used_meta":2691697,"kb_avail":374766512,"statfs":{"total":386547056640,"available":383760908288,"internally_reserved":0,"allocated":29728768,"data_stored":28800588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2756298294},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,15,10,22,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18841600,"num_objects":-2300,"num_object_clones":0,"num_object_copies":-9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004378"},"pg_stats":[{"pgid":"1.f","version":"44'12032","reported_seq":12324,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.618006+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:50.618006+0000","last_peered":"2024-09-10T20:30:50.618006+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:50.618006+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":737280,"num_objects":90,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12032,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12153","reported_seq":12295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.225533+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:49.225533+0000","last_peered":"2024-09-10T20:30:49.225533+0000","last_clean":"2024-09-10T20:30:49.225533+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:49.225533+0000","last_undegraded":"2024-09-10T20:30:49.225533+0000","last_fullsized":"2024-09-10T20:30:49.225533+0000","mapping_epoch":35,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":811008,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8594,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12066","reported_seq":12278,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.158292+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:52.158292+0000","last_peered":"2024-09-10T20:30:52.158292+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:52.158292+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":604776,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12065,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12296","reported_seq":16475,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.474556+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:49.474556+0000","last_peered":"2024-09-10T20:30:49.474556+0000","last_clean":"2024-09-10T20:30:49.474556+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:49.474556+0000","last_undegraded":"2024-09-10T20:30:49.474556+0000","last_fullsized":"2024-09-10T20:30:49.474556+0000","mapping_epoch":37,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":901120,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12045","reported_seq":12372,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.651827+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:50.651827+0000","last_peered":"2024-09-10T20:30:50.651827+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:50.651827+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012896,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1731}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11984","reported_seq":12216,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.657557+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:50.657557+0000","last_peered":"2024-09-10T20:30:50.657557+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:50.657557+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933888,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11951","reported_seq":14305,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:49.147968+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:49.147968+0000","last_peered":"2024-09-10T20:30:49.147968+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:49.147968+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":971936,"num_objects":119,"num_object_clones":0,"num_object_copies":476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-706}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12161","reported_seq":12339,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:48.972623+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:48.972623+0000","last_peered":"2024-09-10T20:30:48.972623+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:48.972623+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1078432,"num_objects":132,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11906","reported_seq":15888,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:50.495487+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:50.495487+0000","last_peered":"2024-09-10T20:30:50.495487+0000","last_clean":"2024-09-10T20:30:50.495487+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:50.495487+0000","last_undegraded":"2024-09-10T20:30:50.495487+0000","last_fullsized":"2024-09-10T20:30:50.495487+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":776784,"num_objects":95,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8459,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12057","reported_seq":16037,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:52.247999+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:52.247999+0000","last_peered":"2024-09-10T20:30:52.247999+0000","last_clean":"2024-09-10T20:30:52.247999+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:52.247999+0000","last_undegraded":"2024-09-10T20:30:52.247999+0000","last_fullsized":"2024-09-10T20:30:52.247999+0000","mapping_epoch":39,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":696320,"num_objects":85,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12016","reported_seq":12299,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.442732+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:50.442732+0000","last_peered":"2024-09-10T20:30:50.442732+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:50.442732+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790256,"num_objects":97,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12013,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12055","reported_seq":12363,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.458595+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:50.458595+0000","last_peered":"2024-09-10T20:30:50.458595+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:50.458595+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":899664,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12054,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1750}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11755","reported_seq":11948,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.191432+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:52.191432+0000","last_peered":"2024-09-10T20:30:52.191432+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:52.191432+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621136,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11990","reported_seq":16050,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.505435+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:49.505435+0000","last_peered":"2024-09-10T20:30:49.505435+0000","last_clean":"2024-09-10T20:30:49.505435+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:49.505435+0000","last_undegraded":"2024-09-10T20:30:49.505435+0000","last_fullsized":"2024-09-10T20:30:49.505435+0000","mapping_epoch":43,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":835584,"num_objects":102,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":942,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12361","reported_seq":12576,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.124809+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:52.124809+0000","last_peered":"2024-09-10T20:30:52.124809+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:52.124809+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2300","ondisk_log_start":"32'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717984,"num_objects":88,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11998","reported_seq":16103,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.375987+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:49.375987+0000","last_peered":"2024-09-10T20:30:49.375987+0000","last_clean":"2024-09-10T20:30:49.375987+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:49.375987+0000","last_undegraded":"2024-09-10T20:30:49.375987+0000","last_fullsized":"2024-09-10T20:30:49.375987+0000","mapping_epoch":41,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2871,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13413064,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152529,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27455488,"data_stored":27455488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160926,"ondisk_log_size":160926,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714928,"kb_used_data":7132,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93656912,"statfs":{"total":96636764160,"available":95904677888,"internally_reserved":0,"allocated":7303168,"data_stored":7071123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704732,"kb_used_data":5760,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93667108,"statfs":{"total":96636764160,"available":95915118592,"internally_reserved":0,"allocated":5898240,"data_stored":5666195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581188,"kb_used_data":8052,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93790652,"statfs":{"total":96636764160,"available":96041627648,"internally_reserved":0,"allocated":8245248,"data_stored":8013203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720000,"kb_used_data":8088,"kb_used_omap":4,"kb_used_meta":711867,"kb_avail":93651840,"statfs":{"total":96636764160,"available":95899484160,"internally_reserved":0,"allocated":8282112,"data_stored":8050067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":728952527},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7868416,"data_stored":7868416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7622656,"data_stored":7622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5279744,"data_stored":5279744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6684672,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:54.193 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:54.782 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:54.816 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-10T20:30:54.138873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13413064,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152529,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160926,"ondisk_log_size":160926,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2720848,"kb_used_data":29032,"kb_used_omap":14,"kb_used_meta":2691697,"kb_avail":374766512,"statfs":{"total":386547056640,"available":383760908288,"internally_reserved":0,"allocated":29728768,"data_stored":28800588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2756298294},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,15,10,22,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18841600,"num_objects":-2300,"num_object_clones":0,"num_object_copies":-9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004378"},"pg_stats":[{"pgid":"1.f","version":"44'12032","reported_seq":12324,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.618006+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:50.618006+0000","last_peered":"2024-09-10T20:30:50.618006+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:50.618006+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":737280,"num_objects":90,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12032,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12153","reported_seq":12295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.225533+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:49.225533+0000","last_peered":"2024-09-10T20:30:49.225533+0000","last_clean":"2024-09-10T20:30:49.225533+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:49.225533+0000","last_undegraded":"2024-09-10T20:30:49.225533+0000","last_fullsized":"2024-09-10T20:30:49.225533+0000","mapping_epoch":35,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":811008,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8594,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12066","reported_seq":12278,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.158292+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:52.158292+0000","last_peered":"2024-09-10T20:30:52.158292+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:52.158292+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":604776,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12065,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12296","reported_seq":16475,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.474556+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:49.474556+0000","last_peered":"2024-09-10T20:30:49.474556+0000","last_clean":"2024-09-10T20:30:49.474556+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:49.474556+0000","last_undegraded":"2024-09-10T20:30:49.474556+0000","last_fullsized":"2024-09-10T20:30:49.474556+0000","mapping_epoch":37,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":901120,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12045","reported_seq":12372,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.651827+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:50.651827+0000","last_peered":"2024-09-10T20:30:50.651827+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:50.651827+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012896,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1731}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'11984","reported_seq":12216,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.657557+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:50.657557+0000","last_peered":"2024-09-10T20:30:50.657557+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:50.657557+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933888,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'11951","reported_seq":14305,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:49.147968+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:49.147968+0000","last_peered":"2024-09-10T20:30:49.147968+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:49.147968+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":971936,"num_objects":119,"num_object_clones":0,"num_object_copies":476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-706}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12161","reported_seq":12339,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:48.972623+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:48.972623+0000","last_peered":"2024-09-10T20:30:48.972623+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:48.972623+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1078432,"num_objects":132,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11906","reported_seq":15888,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:50.495487+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:50.495487+0000","last_peered":"2024-09-10T20:30:50.495487+0000","last_clean":"2024-09-10T20:30:50.495487+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:50.495487+0000","last_undegraded":"2024-09-10T20:30:50.495487+0000","last_fullsized":"2024-09-10T20:30:50.495487+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":776784,"num_objects":95,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8459,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12057","reported_seq":16037,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:52.247999+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:52.247999+0000","last_peered":"2024-09-10T20:30:52.247999+0000","last_clean":"2024-09-10T20:30:52.247999+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:52.247999+0000","last_undegraded":"2024-09-10T20:30:52.247999+0000","last_fullsized":"2024-09-10T20:30:52.247999+0000","mapping_epoch":39,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":696320,"num_objects":85,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12016","reported_seq":12299,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.442732+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:50.442732+0000","last_peered":"2024-09-10T20:30:50.442732+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:50.442732+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790256,"num_objects":97,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12013,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12055","reported_seq":12363,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:50.458595+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:50.458595+0000","last_peered":"2024-09-10T20:30:50.458595+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:50.458595+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":899664,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12054,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1750}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11755","reported_seq":11948,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.191432+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:52.191432+0000","last_peered":"2024-09-10T20:30:52.191432+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:52.191432+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621136,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'11990","reported_seq":16050,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.505435+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:49.505435+0000","last_peered":"2024-09-10T20:30:49.505435+0000","last_clean":"2024-09-10T20:30:49.505435+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:49.505435+0000","last_undegraded":"2024-09-10T20:30:49.505435+0000","last_fullsized":"2024-09-10T20:30:49.505435+0000","mapping_epoch":43,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":835584,"num_objects":102,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":942,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12361","reported_seq":12576,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.124809+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:52.124809+0000","last_peered":"2024-09-10T20:30:52.124809+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:52.124809+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2300","ondisk_log_start":"32'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717984,"num_objects":88,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'11998","reported_seq":16103,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:49.375987+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:49.375987+0000","last_peered":"2024-09-10T20:30:49.375987+0000","last_clean":"2024-09-10T20:30:49.375987+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:49.375987+0000","last_undegraded":"2024-09-10T20:30:49.375987+0000","last_fullsized":"2024-09-10T20:30:49.375987+0000","mapping_epoch":41,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2871,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13413064,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152529,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27455488,"data_stored":27455488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160926,"ondisk_log_size":160926,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714928,"kb_used_data":7132,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93656912,"statfs":{"total":96636764160,"available":95904677888,"internally_reserved":0,"allocated":7303168,"data_stored":7071123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704732,"kb_used_data":5760,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93667108,"statfs":{"total":96636764160,"available":95915118592,"internally_reserved":0,"allocated":5898240,"data_stored":5666195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581188,"kb_used_data":8052,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93790652,"statfs":{"total":96636764160,"available":96041627648,"internally_reserved":0,"allocated":8245248,"data_stored":8013203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720000,"kb_used_data":8088,"kb_used_omap":4,"kb_used_meta":711867,"kb_avail":93651840,"statfs":{"total":96636764160,"available":95899484160,"internally_reserved":0,"allocated":8282112,"data_stored":8050067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":728952527},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7868416,"data_stored":7868416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7622656,"data_stored":7622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5279744,"data_stored":5279744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6684672,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:54.817 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:54.817 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12032", 'reported_seq': 12324, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:50.618006+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:50.618006+0000', 'last_peered': '2024-09-10T20:30:50.618006+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:50.618006+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 737280, 'num_objects': 90, 'num_object_clones': 0, 'num_object_copies': 360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 90, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12032, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1765}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:54.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:54.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12066", 'reported_seq': 12278, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:52.158292+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:52.158292+0000', 'last_peered': '2024-09-10T20:30:52.158292+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:52.158292+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 604776, 'num_objects': 75, 'num_object_clones': 0, 'num_object_copies': 300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 75, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12065, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1833}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:54.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:54.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12045", 'reported_seq': 12372, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:50.651827+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:50.651827+0000', 'last_peered': '2024-09-10T20:30:50.651827+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:50.651827+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1012896, 'num_objects': 124, 'num_object_clones': 0, 'num_object_copies': 496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12043, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1731}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:54.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:54.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'11984", 'reported_seq': 12216, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:50.657557+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:50.657557+0000', 'last_peered': '2024-09-10T20:30:50.657557+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:50.657557+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 933888, 'num_objects': 114, 'num_object_clones': 0, 'num_object_copies': 456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11984, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1743}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'11951", 'reported_seq': 14305, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:49.147968+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:49.147968+0000', 'last_peered': '2024-09-10T20:30:49.147968+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:49.147968+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 971936, 'num_objects': 119, 'num_object_clones': 0, 'num_object_copies': 476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 825, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11949, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1132, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -706}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12161", 'reported_seq': 12339, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:48.972623+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:48.972623+0000', 'last_peered': '2024-09-10T20:30:48.972623+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:48.972623+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1078432, 'num_objects': 132, 'num_object_clones': 0, 'num_object_copies': 528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12159, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1868}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12016", 'reported_seq': 12299, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:50.442732+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:50.442732+0000', 'last_peered': '2024-09-10T20:30:50.442732+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:50.442732+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 790256, 'num_objects': 97, 'num_object_clones': 0, 'num_object_copies': 388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 97, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12013, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1759}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12055", 'reported_seq': 12363, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:50.458595+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:50.458595+0000', 'last_peered': '2024-09-10T20:30:50.458595+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:50.458595+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 899664, 'num_objects': 110, 'num_object_clones': 0, 'num_object_copies': 440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12054, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1750}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:54.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:54.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11755", 'reported_seq': 11948, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:52.191432+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:52.191432+0000', 'last_peered': '2024-09-10T20:30:52.191432+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:52.191432+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 621136, 'num_objects': 76, 'num_object_clones': 0, 'num_object_copies': 304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 76, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11754, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:54.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:54.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12361", 'reported_seq': 12576, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:52.124809+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:52.124809+0000', 'last_peered': '2024-09-10T20:30:52.124809+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:52.124809+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2300", 'ondisk_log_start': "32'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 717984, 'num_objects': 88, 'num_object_clones': 0, 'num_object_copies': 352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 88, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12359, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:54.821 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:30:55.515 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:55.515 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17839/6564 objects degraded (271.770%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":744,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1641,"data_bytes":13413064,"bytes_used":2786148352,"bytes_avail":383760908288,"bytes_total":386547056640,"degraded_objects":17839,"degraded_total":6564,"degraded_ratio":2.7177026203534429,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:30:55.548 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:30:55.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:56.160 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:56.160 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-10T20:30:56.139718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6802120,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153336,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2687404,"kb_used_data":16404,"kb_used_omap":14,"kb_used_meta":2670897,"kb_avail":374799956,"statfs":{"total":386547056640,"available":383795154944,"internally_reserved":0,"allocated":16797696,"data_stored":15869516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2734999094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,13,10,22,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22347776,"num_objects":-2728,"num_object_clones":0,"num_object_copies":-10912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2728,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004556"},"pg_stats":[{"pgid":"1.f","version":"44'12088","reported_seq":12380,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.457188+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:55.457188+0000","last_peered":"2024-09-10T20:30:55.457188+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:55.457188+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12219","reported_seq":12360,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.412894+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:54.412894+0000","last_peered":"2024-09-10T20:30:54.412894+0000","last_clean":"2024-09-10T20:30:54.412894+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:54.412894+0000","last_undegraded":"2024-09-10T20:30:54.412894+0000","last_fullsized":"2024-09-10T20:30:54.412894+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270336,"num_objects":33,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12066","reported_seq":12278,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.158292+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:52.158292+0000","last_peered":"2024-09-10T20:30:52.158292+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:52.158292+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":604776,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12065,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12358","reported_seq":16537,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.501751+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:54.501751+0000","last_peered":"2024-09-10T20:30:54.501751+0000","last_clean":"2024-09-10T20:30:54.501751+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:54.501751+0000","last_undegraded":"2024-09-10T20:30:54.501751+0000","last_fullsized":"2024-09-10T20:30:54.501751+0000","mapping_epoch":37,"log_start":"32'2300","ondisk_log_start":"32'2300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6618,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12119","reported_seq":12448,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.608109+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:55.608109+0000","last_peered":"2024-09-10T20:30:55.608109+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:55.608109+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406688,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12055","reported_seq":12287,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.537251+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:55.537251+0000","last_peered":"2024-09-10T20:30:55.537251+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:55.537251+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":43,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12024","reported_seq":14484,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:54.163174+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:54.163174+0000","last_peered":"2024-09-10T20:30:54.163174+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:54.163174+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373920,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-726}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12236","reported_seq":12412,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:54.065407+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:54.065407+0000","last_peered":"2024-09-10T20:30:54.065407+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:54.065407+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464032,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11965","reported_seq":15947,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:55.572920+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:55.572920+0000","last_peered":"2024-09-10T20:30:55.572920+0000","last_clean":"2024-09-10T20:30:55.572920+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:55.572920+0000","last_undegraded":"2024-09-10T20:30:55.572920+0000","last_fullsized":"2024-09-10T20:30:55.572920+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293456,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12057","reported_seq":16037,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:52.247999+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:52.247999+0000","last_peered":"2024-09-10T20:30:52.247999+0000","last_clean":"2024-09-10T20:30:52.247999+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:52.247999+0000","last_undegraded":"2024-09-10T20:30:52.247999+0000","last_fullsized":"2024-09-10T20:30:52.247999+0000","mapping_epoch":39,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":696320,"num_objects":85,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12074","reported_seq":12357,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.650675+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:55.650675+0000","last_peered":"2024-09-10T20:30:55.650675+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:55.650675+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315120,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12123","reported_seq":12431,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.677120+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:55.677120+0000","last_peered":"2024-09-10T20:30:55.677120+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:55.677120+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342608,"num_objects":42,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1818}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11755","reported_seq":11948,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.191432+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:52.191432+0000","last_peered":"2024-09-10T20:30:52.191432+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:52.191432+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621136,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12056","reported_seq":16116,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.437139+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:54.437139+0000","last_peered":"2024-09-10T20:30:54.437139+0000","last_clean":"2024-09-10T20:30:54.437139+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:54.437139+0000","last_undegraded":"2024-09-10T20:30:54.437139+0000","last_fullsized":"2024-09-10T20:30:54.437139+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294912,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12361","reported_seq":12576,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:52.124809+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:52.124809+0000","last_peered":"2024-09-10T20:30:52.124809+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:52.124809+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2300","ondisk_log_start":"32'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717984,"num_objects":88,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12077","reported_seq":16182,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.305367+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:54.305367+0000","last_peered":"2024-09-10T20:30:54.305367+0000","last_clean":"2024-09-10T20:30:54.305367+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:54.305367+0000","last_undegraded":"2024-09-10T20:30:54.305367+0000","last_fullsized":"2024-09-10T20:30:54.305367+0000","mapping_epoch":41,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376832,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6802120,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153336,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14528512,"data_stored":14528512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698244,"kb_used_data":2928,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93673596,"statfs":{"total":96636764160,"available":95921762304,"internally_reserved":0,"allocated":2998272,"data_stored":2766227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":711978704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704732,"kb_used_data":5760,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93667108,"statfs":{"total":96636764160,"available":95915118592,"internally_reserved":0,"allocated":5898240,"data_stored":5666195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581124,"kb_used_data":3820,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93790716,"statfs":{"total":96636764160,"available":96041693184,"internally_reserved":0,"allocated":3911680,"data_stored":3679635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703304,"kb_used_data":3896,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93668536,"statfs":{"total":96636764160,"available":95916580864,"internally_reserved":0,"allocated":3989504,"data_stored":3757459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3584000,"data_stored":3584000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3317760,"data_stored":3317760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5279744,"data_stored":5279744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2347008,"data_stored":2347008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:58.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Removed 97184 objects 2024-09-10T20:30:58.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi103.stdout:Clean up completed and total clean up time :435.563 2024-09-10T20:30:59.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:59.505 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:59.517 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-10T20:30:58.140443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4696776,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153593,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2687328,"kb_used_data":12180,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374800032,"statfs":{"total":386547056640,"available":383795232768,"internally_reserved":0,"allocated":12472320,"data_stored":11544140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,15,7,28,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19251200,"num_objects":-2350,"num_object_clones":0,"num_object_copies":-9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004656"},"pg_stats":[{"pgid":"1.f","version":"44'12088","reported_seq":12380,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.457188+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:55.457188+0000","last_peered":"2024-09-10T20:30:55.457188+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:55.457188+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12219","reported_seq":12360,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.412894+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:54.412894+0000","last_peered":"2024-09-10T20:30:54.412894+0000","last_clean":"2024-09-10T20:30:54.412894+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:54.412894+0000","last_undegraded":"2024-09-10T20:30:54.412894+0000","last_fullsized":"2024-09-10T20:30:54.412894+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270336,"num_objects":33,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12128","reported_seq":12340,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.318168+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:57.318168+0000","last_peered":"2024-09-10T20:30:57.318168+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:57.318168+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96872,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12127,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1895}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12358","reported_seq":16537,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.501751+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:54.501751+0000","last_peered":"2024-09-10T20:30:54.501751+0000","last_clean":"2024-09-10T20:30:54.501751+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:54.501751+0000","last_undegraded":"2024-09-10T20:30:54.501751+0000","last_fullsized":"2024-09-10T20:30:54.501751+0000","mapping_epoch":37,"log_start":"32'2300","ondisk_log_start":"32'2300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6618,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12119","reported_seq":12448,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.608109+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:55.608109+0000","last_peered":"2024-09-10T20:30:55.608109+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:55.608109+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406688,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12055","reported_seq":12287,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.537251+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:55.537251+0000","last_peered":"2024-09-10T20:30:55.537251+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:55.537251+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":43,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12024","reported_seq":14484,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:54.163174+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:54.163174+0000","last_peered":"2024-09-10T20:30:54.163174+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:54.163174+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373920,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-726}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12236","reported_seq":12412,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:54.065407+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:54.065407+0000","last_peered":"2024-09-10T20:30:54.065407+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:54.065407+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464032,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11965","reported_seq":15947,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:55.572920+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:55.572920+0000","last_peered":"2024-09-10T20:30:55.572920+0000","last_clean":"2024-09-10T20:30:55.572920+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:55.572920+0000","last_undegraded":"2024-09-10T20:30:55.572920+0000","last_fullsized":"2024-09-10T20:30:55.572920+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293456,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12128","reported_seq":16108,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:57.080973+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:57.080973+0000","last_peered":"2024-09-10T20:30:57.080973+0000","last_clean":"2024-09-10T20:30:57.080973+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:57.080973+0000","last_undegraded":"2024-09-10T20:30:57.080973+0000","last_fullsized":"2024-09-10T20:30:57.080973+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12074","reported_seq":12357,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.650675+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:55.650675+0000","last_peered":"2024-09-10T20:30:55.650675+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:55.650675+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315120,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12123","reported_seq":12431,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.677120+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:55.677120+0000","last_peered":"2024-09-10T20:30:55.677120+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:55.677120+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342608,"num_objects":42,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1818}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11813","reported_seq":12006,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.149021+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:57.149021+0000","last_peered":"2024-09-10T20:30:57.149021+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:57.149021+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146000,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11812,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12056","reported_seq":16116,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.437139+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:54.437139+0000","last_peered":"2024-09-10T20:30:54.437139+0000","last_clean":"2024-09-10T20:30:54.437139+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:54.437139+0000","last_undegraded":"2024-09-10T20:30:54.437139+0000","last_fullsized":"2024-09-10T20:30:54.437139+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294912,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12427","reported_seq":12642,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.197603+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:57.197603+0000","last_peered":"2024-09-10T20:30:57.197603+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:57.197603+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177312,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12077","reported_seq":16182,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.305367+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:54.305367+0000","last_peered":"2024-09-10T20:30:54.305367+0000","last_clean":"2024-09-10T20:30:54.305367+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:54.305367+0000","last_undegraded":"2024-09-10T20:30:54.305367+0000","last_fullsized":"2024-09-10T20:30:54.305367+0000","mapping_epoch":41,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376832,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":4696776,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153593,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10215424,"data_stored":10215424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698244,"kb_used_data":2928,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93673596,"statfs":{"total":96636764160,"available":95921762304,"internally_reserved":0,"allocated":2998272,"data_stored":2766227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":711978704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704656,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93667184,"statfs":{"total":96636764160,"available":95915196416,"internally_reserved":0,"allocated":1572864,"data_stored":1340819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581124,"kb_used_data":3820,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93790716,"statfs":{"total":96636764160,"available":96041693184,"internally_reserved":0,"allocated":3911680,"data_stored":3679635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703304,"kb_used_data":3896,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93668536,"statfs":{"total":96636764160,"available":95916580864,"internally_reserved":0,"allocated":3989504,"data_stored":3757459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3584000,"data_stored":3584000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3317760,"data_stored":3317760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":966656,"data_stored":966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2347008,"data_stored":2347008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:59.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:30:59.835 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:30:59.836 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:30:59.848 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-10T20:30:58.140443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4696776,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153593,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2687328,"kb_used_data":12180,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374800032,"statfs":{"total":386547056640,"available":383795232768,"internally_reserved":0,"allocated":12472320,"data_stored":11544140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,15,7,28,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19251200,"num_objects":-2350,"num_object_clones":0,"num_object_copies":-9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004656"},"pg_stats":[{"pgid":"1.f","version":"44'12088","reported_seq":12380,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.457188+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:55.457188+0000","last_peered":"2024-09-10T20:30:55.457188+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:55.457188+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12219","reported_seq":12360,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.412894+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:54.412894+0000","last_peered":"2024-09-10T20:30:54.412894+0000","last_clean":"2024-09-10T20:30:54.412894+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:54.412894+0000","last_undegraded":"2024-09-10T20:30:54.412894+0000","last_fullsized":"2024-09-10T20:30:54.412894+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270336,"num_objects":33,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12128","reported_seq":12340,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.318168+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:57.318168+0000","last_peered":"2024-09-10T20:30:57.318168+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:57.318168+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96872,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12127,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1895}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12358","reported_seq":16537,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.501751+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:54.501751+0000","last_peered":"2024-09-10T20:30:54.501751+0000","last_clean":"2024-09-10T20:30:54.501751+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:54.501751+0000","last_undegraded":"2024-09-10T20:30:54.501751+0000","last_fullsized":"2024-09-10T20:30:54.501751+0000","mapping_epoch":37,"log_start":"32'2300","ondisk_log_start":"32'2300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6618,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12119","reported_seq":12448,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.608109+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:55.608109+0000","last_peered":"2024-09-10T20:30:55.608109+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:55.608109+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406688,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12055","reported_seq":12287,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.537251+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:55.537251+0000","last_peered":"2024-09-10T20:30:55.537251+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:55.537251+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":43,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12024","reported_seq":14484,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:54.163174+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:54.163174+0000","last_peered":"2024-09-10T20:30:54.163174+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:54.163174+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373920,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-726}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12236","reported_seq":12412,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:54.065407+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:54.065407+0000","last_peered":"2024-09-10T20:30:54.065407+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:54.065407+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464032,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11965","reported_seq":15947,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:55.572920+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:55.572920+0000","last_peered":"2024-09-10T20:30:55.572920+0000","last_clean":"2024-09-10T20:30:55.572920+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:55.572920+0000","last_undegraded":"2024-09-10T20:30:55.572920+0000","last_fullsized":"2024-09-10T20:30:55.572920+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293456,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12128","reported_seq":16108,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:57.080973+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:57.080973+0000","last_peered":"2024-09-10T20:30:57.080973+0000","last_clean":"2024-09-10T20:30:57.080973+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:57.080973+0000","last_undegraded":"2024-09-10T20:30:57.080973+0000","last_fullsized":"2024-09-10T20:30:57.080973+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12074","reported_seq":12357,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.650675+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:55.650675+0000","last_peered":"2024-09-10T20:30:55.650675+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:55.650675+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315120,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12123","reported_seq":12431,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.677120+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:55.677120+0000","last_peered":"2024-09-10T20:30:55.677120+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:55.677120+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342608,"num_objects":42,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1818}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11813","reported_seq":12006,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.149021+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:57.149021+0000","last_peered":"2024-09-10T20:30:57.149021+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:57.149021+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146000,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11812,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12056","reported_seq":16116,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.437139+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:54.437139+0000","last_peered":"2024-09-10T20:30:54.437139+0000","last_clean":"2024-09-10T20:30:54.437139+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:54.437139+0000","last_undegraded":"2024-09-10T20:30:54.437139+0000","last_fullsized":"2024-09-10T20:30:54.437139+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294912,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12427","reported_seq":12642,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.197603+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:57.197603+0000","last_peered":"2024-09-10T20:30:57.197603+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:57.197603+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177312,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12077","reported_seq":16182,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:54.305367+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:54.305367+0000","last_peered":"2024-09-10T20:30:54.305367+0000","last_clean":"2024-09-10T20:30:54.305367+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:54.305367+0000","last_undegraded":"2024-09-10T20:30:54.305367+0000","last_fullsized":"2024-09-10T20:30:54.305367+0000","mapping_epoch":41,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376832,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":4696776,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153593,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10215424,"data_stored":10215424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698244,"kb_used_data":2928,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93673596,"statfs":{"total":96636764160,"available":95921762304,"internally_reserved":0,"allocated":2998272,"data_stored":2766227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":711978704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704656,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93667184,"statfs":{"total":96636764160,"available":95915196416,"internally_reserved":0,"allocated":1572864,"data_stored":1340819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581124,"kb_used_data":3820,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93790716,"statfs":{"total":96636764160,"available":96041693184,"internally_reserved":0,"allocated":3911680,"data_stored":3679635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703304,"kb_used_data":3896,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93668536,"statfs":{"total":96636764160,"available":95916580864,"internally_reserved":0,"allocated":3989504,"data_stored":3757459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3584000,"data_stored":3584000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3317760,"data_stored":3317760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":966656,"data_stored":966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2347008,"data_stored":2347008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:30:59.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:30:59.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12088", 'reported_seq': 12380, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:55.457188+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:55.457188+0000', 'last_peered': '2024-09-10T20:30:55.457188+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:55.457188+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 278528, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 34, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12088, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1821}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12128", 'reported_seq': 12340, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:57.318168+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:30:57.318168+0000', 'last_peered': '2024-09-10T20:30:57.318168+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:30:57.318168+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96872, 'num_objects': 13, 'num_object_clones': 0, 'num_object_copies': 52, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 13, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12127, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1895}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12119", 'reported_seq': 12448, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:55.608109+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:55.608109+0000', 'last_peered': '2024-09-10T20:30:55.608109+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:55.608109+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 406688, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 50, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12117, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1805}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12055", 'reported_seq': 12287, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:55.537251+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:55.537251+0000', 'last_peered': '2024-09-10T20:30:55.537251+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:55.537251+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 352256, 'num_objects': 43, 'num_object_clones': 0, 'num_object_copies': 172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 43, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12055, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1814}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:30:59.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'12024", 'reported_seq': 14484, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:54.163174+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:54.163174+0000', 'last_peered': '2024-09-10T20:30:54.163174+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:54.163174+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 373920, 'num_objects': 46, 'num_object_clones': 0, 'num_object_copies': 184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 772, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 46, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12022, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1185, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -726}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12236", 'reported_seq': 12412, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:54.065407+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:54.065407+0000', 'last_peered': '2024-09-10T20:30:54.065407+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:54.065407+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "32'2200", 'ondisk_log_start': "32'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 464032, 'num_objects': 57, 'num_object_clones': 0, 'num_object_copies': 228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 57, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12234, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1943}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12074", 'reported_seq': 12357, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:55.650675+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:55.650675+0000', 'last_peered': '2024-09-10T20:30:55.650675+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:55.650675+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 315120, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12071, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1817}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12123", 'reported_seq': 12431, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:55.677120+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:55.677120+0000', 'last_peered': '2024-09-10T20:30:55.677120+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:55.677120+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 342608, 'num_objects': 42, 'num_object_clones': 0, 'num_object_copies': 168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 42, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11813", 'reported_seq': 12006, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:57.149021+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:30:57.149021+0000', 'last_peered': '2024-09-10T20:30:57.149021+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:30:57.149021+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 146000, 'num_objects': 18, 'num_object_clones': 0, 'num_object_copies': 72, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11812, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:30:59.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12427", 'reported_seq': 12642, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:57.197603+0000', 'last_change': '2024-09-10T20:20:16.117431+0000', 'last_active': '2024-09-10T20:30:57.197603+0000', 'last_peered': '2024-09-10T20:30:57.197603+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:30:57.197603+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2400", 'ondisk_log_start': "32'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 177312, 'num_objects': 22, 'num_object_clones': 0, 'num_object_copies': 88, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 22, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:30:59.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:00.221 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:00.221 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17559/1260 objects degraded (1393.571%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":748,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":315,"data_bytes":2550472,"bytes_used":2748968960,"bytes_avail":383798087680,"bytes_total":386547056640,"degraded_objects":17559,"degraded_total":1260,"degraded_ratio":13.935714285714285,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":355,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:00.235 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:31:00.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:00.533 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:00.533 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:00.545 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-10T20:31:00.140912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2550472,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153855,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160752,"ondisk_log_size":160752,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2684540,"kb_used_data":5240,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374802820,"statfs":{"total":386547056640,"available":383798087680,"internally_reserved":0,"allocated":5365760,"data_stored":4437580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,14,2,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19333120,"num_objects":-2360,"num_object_clones":0,"num_object_copies":-9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004309"},"pg_stats":[{"pgid":"1.f","version":"44'12088","reported_seq":12380,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.457188+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:55.457188+0000","last_peered":"2024-09-10T20:30:55.457188+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:55.457188+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12394,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.313321+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:58.313321+0000","last_peered":"2024-09-10T20:30:58.313321+0000","last_clean":"2024-09-10T20:30:58.313321+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:58.313321+0000","last_undegraded":"2024-09-10T20:30:58.313321+0000","last_fullsized":"2024-09-10T20:30:58.313321+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12128","reported_seq":12340,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.318168+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:30:57.318168+0000","last_peered":"2024-09-10T20:30:57.318168+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:30:57.318168+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96872,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12127,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1895}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.260855+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:58.260855+0000","last_peered":"2024-09-10T20:30:58.260855+0000","last_clean":"2024-09-10T20:30:58.260855+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:58.260855+0000","last_undegraded":"2024-09-10T20:30:58.260855+0000","last_fullsized":"2024-09-10T20:30:58.260855+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12119","reported_seq":12448,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.608109+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:55.608109+0000","last_peered":"2024-09-10T20:30:55.608109+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:55.608109+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406688,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12055","reported_seq":12287,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.537251+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:55.537251+0000","last_peered":"2024-09-10T20:30:55.537251+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:55.537251+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":43,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":14982,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:59.224268+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:59.224268+0000","last_peered":"2024-09-10T20:30:59.224268+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:59.224268+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":545,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-543}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12469,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.287893+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:58.287893+0000","last_peered":"2024-09-10T20:30:58.287893+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:58.287893+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'11965","reported_seq":15947,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:55.572920+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:55.572920+0000","last_peered":"2024-09-10T20:30:55.572920+0000","last_clean":"2024-09-10T20:30:55.572920+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:55.572920+0000","last_undegraded":"2024-09-10T20:30:55.572920+0000","last_fullsized":"2024-09-10T20:30:55.572920+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293456,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12128","reported_seq":16108,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:57.080973+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:30:57.080973+0000","last_peered":"2024-09-10T20:30:57.080973+0000","last_clean":"2024-09-10T20:30:57.080973+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:30:57.080973+0000","last_undegraded":"2024-09-10T20:30:57.080973+0000","last_fullsized":"2024-09-10T20:30:57.080973+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12074","reported_seq":12357,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.650675+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:55.650675+0000","last_peered":"2024-09-10T20:30:55.650675+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:55.650675+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315120,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12123","reported_seq":12431,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:55.677120+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:55.677120+0000","last_peered":"2024-09-10T20:30:55.677120+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:55.677120+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342608,"num_objects":42,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1818}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11813","reported_seq":12006,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.149021+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:30:57.149021+0000","last_peered":"2024-09-10T20:30:57.149021+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:30:57.149021+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146000,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11812,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16152,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.146591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:58.146591+0000","last_peered":"2024-09-10T20:30:58.146591+0000","last_clean":"2024-09-10T20:30:58.146591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:58.146591+0000","last_undegraded":"2024-09-10T20:30:58.146591+0000","last_fullsized":"2024-09-10T20:30:58.146591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12427","reported_seq":12642,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:57.197603+0000","last_change":"2024-09-10T20:20:16.117431+0000","last_active":"2024-09-10T20:30:57.197603+0000","last_peered":"2024-09-10T20:30:57.197603+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:30:57.197603+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177312,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16228,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.318689+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:58.318689+0000","last_peered":"2024-09-10T20:30:58.318689+0000","last_clean":"2024-09-10T20:30:58.318689+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:58.318689+0000","last_undegraded":"2024-09-10T20:30:58.318689+0000","last_fullsized":"2024-09-10T20:30:58.318689+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":2550472,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153855,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3371008,"data_stored":3371008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160752,"ondisk_log_size":160752,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698244,"kb_used_data":2928,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93673596,"statfs":{"total":96636764160,"available":95921762304,"internally_reserved":0,"allocated":2998272,"data_stored":2766227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":711978704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704656,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93667184,"statfs":{"total":96636764160,"available":95915196416,"internally_reserved":0,"allocated":1572864,"data_stored":1340819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581868,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789972,"statfs":{"total":96636764160,"available":96040931328,"internally_reserved":0,"allocated":421888,"data_stored":189843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699772,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672068,"statfs":{"total":96636764160,"available":95920197632,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":966656,"data_stored":966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2347008,"data_stored":2347008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:03.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:03.881 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:03.893 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-09-10T20:31:03.208895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101040,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154155,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2684960,"kb_used_data":1504,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374802400,"statfs":{"total":386547056640,"available":383797657600,"internally_reserved":0,"allocated":1540096,"data_stored":611916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2747778614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13312024,"num_objects":-1626,"num_object_clones":0,"num_object_copies":-6504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1626,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.070021"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12414,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.364023+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:58.364023+0000","last_peered":"2024-09-10T20:30:58.364023+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:58.364023+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12394,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.313321+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:58.313321+0000","last_peered":"2024-09-10T20:30:58.313321+0000","last_clean":"2024-09-10T20:30:58.313321+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:58.313321+0000","last_undegraded":"2024-09-10T20:30:58.313321+0000","last_fullsized":"2024-09-10T20:30:58.313321+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12355,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.209502+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:02.209502+0000","last_peered":"2024-09-10T20:31:02.209502+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:02.209502+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.260855+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:58.260855+0000","last_peered":"2024-09-10T20:30:58.260855+0000","last_clean":"2024-09-10T20:30:58.260855+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:58.260855+0000","last_undegraded":"2024-09-10T20:30:58.260855+0000","last_fullsized":"2024-09-10T20:30:58.260855+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12497,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.224744+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:58.224744+0000","last_peered":"2024-09-10T20:30:58.224744+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:58.224744+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12330,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.294586+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:58.294586+0000","last_peered":"2024-09-10T20:30:58.294586+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:58.294586+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":14982,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:59.224268+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:59.224268+0000","last_peered":"2024-09-10T20:30:59.224268+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:59.224268+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":545,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-543}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12469,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.287893+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:58.287893+0000","last_peered":"2024-09-10T20:30:58.287893+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:58.287893+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15982,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.298013+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:58.298013+0000","last_peered":"2024-09-10T20:30:58.298013+0000","last_clean":"2024-09-10T20:30:58.298013+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:58.298013+0000","last_undegraded":"2024-09-10T20:30:58.298013+0000","last_fullsized":"2024-09-10T20:30:58.298013+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-10T20:31:02.211011+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:02.211011+0000","last_peered":"2024-09-10T20:31:02.211011+0000","last_clean":"2024-09-10T20:31:02.211011+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:02.211011+0000","last_undegraded":"2024-09-10T20:31:02.211011+0000","last_fullsized":"2024-09-10T20:31:02.211011+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12393,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.354293+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:58.354293+0000","last_peered":"2024-09-10T20:30:58.354293+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:58.354293+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12472,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.362461+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:58.362461+0000","last_peered":"2024-09-10T20:30:58.362461+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:58.362461+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12025,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.209571+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:02.209571+0000","last_peered":"2024-09-10T20:31:02.209571+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:02.209571+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16152,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.146591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:58.146591+0000","last_peered":"2024-09-10T20:30:58.146591+0000","last_clean":"2024-09-10T20:30:58.146591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:58.146591+0000","last_undegraded":"2024-09-10T20:30:58.146591+0000","last_fullsized":"2024-09-10T20:30:58.146591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12909,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.369538+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:02.369538+0000","last_peered":"2024-09-10T20:31:02.369538+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:02.369538+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1786}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16228,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.318689+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:58.318689+0000","last_peered":"2024-09-10T20:30:58.318689+0000","last_clean":"2024-09-10T20:30:58.318689+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:58.318689+0000","last_undegraded":"2024-09-10T20:30:58.318689+0000","last_fullsized":"2024-09-10T20:30:58.318689+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101040,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154155,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699836,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93672004,"statfs":{"total":96636764160,"available":95920132096,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703484,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668356,"statfs":{"total":96636764160,"available":95916396544,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581868,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789972,"statfs":{"total":96636764160,"available":96040931328,"internally_reserved":0,"allocated":421888,"data_stored":189843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699772,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672068,"statfs":{"total":96636764160,"available":95920197632,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:03.894 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:04.231 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:04.231 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:04.244 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-10T20:31:04.141609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101040,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154155,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2684960,"kb_used_data":1504,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374802400,"statfs":{"total":386547056640,"available":383797657600,"internally_reserved":0,"allocated":1540096,"data_stored":611916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2747778614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6701080,"num_objects":-819,"num_object_clones":0,"num_object_copies":-3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":819,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":349,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001891"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12414,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.364023+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:58.364023+0000","last_peered":"2024-09-10T20:30:58.364023+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:58.364023+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12394,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.313321+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:58.313321+0000","last_peered":"2024-09-10T20:30:58.313321+0000","last_clean":"2024-09-10T20:30:58.313321+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:58.313321+0000","last_undegraded":"2024-09-10T20:30:58.313321+0000","last_fullsized":"2024-09-10T20:30:58.313321+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12355,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.209502+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:02.209502+0000","last_peered":"2024-09-10T20:31:02.209502+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:02.209502+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.260855+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:58.260855+0000","last_peered":"2024-09-10T20:30:58.260855+0000","last_clean":"2024-09-10T20:30:58.260855+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:58.260855+0000","last_undegraded":"2024-09-10T20:30:58.260855+0000","last_fullsized":"2024-09-10T20:30:58.260855+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12497,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.224744+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:58.224744+0000","last_peered":"2024-09-10T20:30:58.224744+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:58.224744+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12330,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.294586+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:58.294586+0000","last_peered":"2024-09-10T20:30:58.294586+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:58.294586+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":14982,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:59.224268+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:59.224268+0000","last_peered":"2024-09-10T20:30:59.224268+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:59.224268+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":545,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-543}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12469,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.287893+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:58.287893+0000","last_peered":"2024-09-10T20:30:58.287893+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:58.287893+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15982,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.298013+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:58.298013+0000","last_peered":"2024-09-10T20:30:58.298013+0000","last_clean":"2024-09-10T20:30:58.298013+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:58.298013+0000","last_undegraded":"2024-09-10T20:30:58.298013+0000","last_fullsized":"2024-09-10T20:30:58.298013+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-10T20:31:02.211011+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:02.211011+0000","last_peered":"2024-09-10T20:31:02.211011+0000","last_clean":"2024-09-10T20:31:02.211011+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:02.211011+0000","last_undegraded":"2024-09-10T20:31:02.211011+0000","last_fullsized":"2024-09-10T20:31:02.211011+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12393,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.354293+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:58.354293+0000","last_peered":"2024-09-10T20:30:58.354293+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:58.354293+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12472,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.362461+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:58.362461+0000","last_peered":"2024-09-10T20:30:58.362461+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:58.362461+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12025,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.209571+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:02.209571+0000","last_peered":"2024-09-10T20:31:02.209571+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:02.209571+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16152,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.146591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:58.146591+0000","last_peered":"2024-09-10T20:30:58.146591+0000","last_clean":"2024-09-10T20:30:58.146591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:58.146591+0000","last_undegraded":"2024-09-10T20:30:58.146591+0000","last_fullsized":"2024-09-10T20:30:58.146591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12909,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.369538+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:02.369538+0000","last_peered":"2024-09-10T20:31:02.369538+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:02.369538+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1786}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16228,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.318689+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:58.318689+0000","last_peered":"2024-09-10T20:30:58.318689+0000","last_clean":"2024-09-10T20:30:58.318689+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:58.318689+0000","last_undegraded":"2024-09-10T20:30:58.318689+0000","last_fullsized":"2024-09-10T20:30:58.318689+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101040,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154155,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699836,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93672004,"statfs":{"total":96636764160,"available":95920132096,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703484,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668356,"statfs":{"total":96636764160,"available":95916396544,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581868,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789972,"statfs":{"total":96636764160,"available":96040931328,"internally_reserved":0,"allocated":421888,"data_stored":189843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699772,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672068,"statfs":{"total":96636764160,"available":95920197632,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:04.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:04.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12414, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:58.364023+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:30:58.364023+0000', 'last_peered': '2024-09-10T20:30:58.364023+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:30:58.364023+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:04.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:04.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12355, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:02.209502+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:02.209502+0000', 'last_peered': '2024-09-10T20:31:02.209502+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:02.209502+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12497, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:58.224744+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:30:58.224744+0000', 'last_peered': '2024-09-10T20:30:58.224744+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:30:58.224744+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 12330, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:58.294586+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:30:58.294586+0000', 'last_peered': '2024-09-10T20:30:58.294586+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:30:58.294586+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1857}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "44'12068", 'reported_seq': 14982, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:59.224268+0000', 'last_change': '2024-09-10T20:29:30.402546+0000', 'last_active': '2024-09-10T20:30:59.224268+0000', 'last_peered': '2024-09-10T20:30:59.224268+0000', 'last_clean': '2024-09-10T20:19:06.411270+0000', 'last_became_active': '2024-09-10T20:20:16.144154+0000', 'last_became_peered': '2024-09-10T20:20:16.144154+0000', 'last_unstale': '2024-09-10T20:30:59.224268+0000', 'last_undegraded': '2024-09-10T20:20:15.496308+0000', 'last_fullsized': '2024-09-10T20:20:15.454573+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 545, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12066, 'num_write_kb': 48268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1412, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -543}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12291", 'reported_seq': 12469, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:58.287893+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:30:58.287893+0000', 'last_peered': '2024-09-10T20:30:58.287893+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:30:58.287893+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "32'2200", 'ondisk_log_start': "32'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12289, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:04.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12393, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:58.354293+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:30:58.354293+0000', 'last_peered': '2024-09-10T20:30:58.354293+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:30:58.354293+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:04.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:04.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12472, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:30:58.362461+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:30:58.362461+0000', 'last_peered': '2024-09-10T20:30:58.362461+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:30:58.362461+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:04.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:04.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12025, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:02.209571+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:31:02.209571+0000', 'last_peered': '2024-09-10T20:31:02.209571+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:02.209571+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:04.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:31:04.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12447", 'reported_seq': 12909, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:02.369538+0000', 'last_change': '2024-09-10T20:31:01.583331+0000', 'last_active': '2024-09-10T20:31:02.369538+0000', 'last_peered': '2024-09-10T20:31:02.369538+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:31:02.369538+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2400", 'ondisk_log_start': "32'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1788, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 178, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:04.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:04.649 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:04.649 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17437/60 objects degraded (29061.667%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":753,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":15,"data_bytes":101040,"bytes_used":2749399040,"bytes_avail":383797657600,"bytes_total":386547056640,"degraded_objects":17437,"degraded_total":60,"degraded_ratio":290.61666666666667,"recovering_objects_per_sec":43,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":349,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":102},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:04.663 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:31:04.663 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:04.986 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:04.998 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-10T20:31:04.141609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101040,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154155,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2684960,"kb_used_data":1504,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374802400,"statfs":{"total":386547056640,"available":383797657600,"internally_reserved":0,"allocated":1540096,"data_stored":611916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2747778614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6701080,"num_objects":-819,"num_object_clones":0,"num_object_copies":-3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":819,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":349,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001891"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12414,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.364023+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:30:58.364023+0000","last_peered":"2024-09-10T20:30:58.364023+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:30:58.364023+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12394,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.313321+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:30:58.313321+0000","last_peered":"2024-09-10T20:30:58.313321+0000","last_clean":"2024-09-10T20:30:58.313321+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:30:58.313321+0000","last_undegraded":"2024-09-10T20:30:58.313321+0000","last_fullsized":"2024-09-10T20:30:58.313321+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12355,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.209502+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:02.209502+0000","last_peered":"2024-09-10T20:31:02.209502+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:02.209502+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.260855+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:30:58.260855+0000","last_peered":"2024-09-10T20:30:58.260855+0000","last_clean":"2024-09-10T20:30:58.260855+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:30:58.260855+0000","last_undegraded":"2024-09-10T20:30:58.260855+0000","last_fullsized":"2024-09-10T20:30:58.260855+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12497,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.224744+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:30:58.224744+0000","last_peered":"2024-09-10T20:30:58.224744+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:30:58.224744+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12330,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.294586+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:30:58.294586+0000","last_peered":"2024-09-10T20:30:58.294586+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:30:58.294586+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":14982,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:59.224268+0000","last_change":"2024-09-10T20:29:30.402546+0000","last_active":"2024-09-10T20:30:59.224268+0000","last_peered":"2024-09-10T20:30:59.224268+0000","last_clean":"2024-09-10T20:19:06.411270+0000","last_became_active":"2024-09-10T20:20:16.144154+0000","last_became_peered":"2024-09-10T20:20:16.144154+0000","last_unstale":"2024-09-10T20:30:59.224268+0000","last_undegraded":"2024-09-10T20:20:15.496308+0000","last_fullsized":"2024-09-10T20:20:15.454573+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":545,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-543}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12469,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.287893+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:30:58.287893+0000","last_peered":"2024-09-10T20:30:58.287893+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:30:58.287893+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15982,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.298013+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:30:58.298013+0000","last_peered":"2024-09-10T20:30:58.298013+0000","last_clean":"2024-09-10T20:30:58.298013+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:30:58.298013+0000","last_undegraded":"2024-09-10T20:30:58.298013+0000","last_fullsized":"2024-09-10T20:30:58.298013+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-10T20:31:02.211011+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:02.211011+0000","last_peered":"2024-09-10T20:31:02.211011+0000","last_clean":"2024-09-10T20:31:02.211011+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:02.211011+0000","last_undegraded":"2024-09-10T20:31:02.211011+0000","last_fullsized":"2024-09-10T20:31:02.211011+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12393,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.354293+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:30:58.354293+0000","last_peered":"2024-09-10T20:30:58.354293+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:30:58.354293+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12472,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:30:58.362461+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:30:58.362461+0000","last_peered":"2024-09-10T20:30:58.362461+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:30:58.362461+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12025,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.209571+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:02.209571+0000","last_peered":"2024-09-10T20:31:02.209571+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:02.209571+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16152,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.146591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:30:58.146591+0000","last_peered":"2024-09-10T20:30:58.146591+0000","last_clean":"2024-09-10T20:30:58.146591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:30:58.146591+0000","last_undegraded":"2024-09-10T20:30:58.146591+0000","last_fullsized":"2024-09-10T20:30:58.146591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12909,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:02.369538+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:02.369538+0000","last_peered":"2024-09-10T20:31:02.369538+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:02.369538+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1786}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16228,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-10T20:30:58.318689+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:30:58.318689+0000","last_peered":"2024-09-10T20:30:58.318689+0000","last_clean":"2024-09-10T20:30:58.318689+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:30:58.318689+0000","last_undegraded":"2024-09-10T20:30:58.318689+0000","last_fullsized":"2024-09-10T20:30:58.318689+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101040,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154155,"num_write_kb":530491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699836,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93672004,"statfs":{"total":96636764160,"available":95920132096,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703484,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668356,"statfs":{"total":96636764160,"available":95916396544,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581868,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789972,"statfs":{"total":96636764160,"available":96040931328,"internally_reserved":0,"allocated":421888,"data_stored":189843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699772,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672068,"statfs":{"total":96636764160,"available":95920197632,"internally_reserved":0,"allocated":372736,"data_stored":140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:08.000 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:08.311 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:08.323 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-10T20:31:08.142407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676716,"kb_used_data":1560,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374810644,"statfs":{"total":386547056640,"available":383806099456,"internally_reserved":0,"allocated":1597440,"data_stored":630848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2462904,"num_objects":-302,"num_object_clones":0,"num_object_copies":-1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11766,"num_write_kb":-48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001494"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12418,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211994+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:03.211994+0000","last_peered":"2024-09-10T20:31:03.211994+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:03.211994+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12398,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210801+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:03.210801+0000","last_peered":"2024-09-10T20:31:03.210801+0000","last_clean":"2024-09-10T20:31:03.210801+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:03.210801+0000","last_undegraded":"2024-09-10T20:31:03.210801+0000","last_fullsized":"2024-09-10T20:31:03.210801+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12357,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.214988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:03.214988+0000","last_peered":"2024-09-10T20:31:03.214988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:03.214988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16589,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210258+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:03.210258+0000","last_peered":"2024-09-10T20:31:03.210258+0000","last_clean":"2024-09-10T20:31:03.210258+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:03.210258+0000","last_undegraded":"2024-09-10T20:31:03.210258+0000","last_fullsized":"2024-09-10T20:31:03.210258+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12501,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210284+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:03.210284+0000","last_peered":"2024-09-10T20:31:03.210284+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:03.210284+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12334,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210790+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:03.210790+0000","last_peered":"2024-09-10T20:31:03.210790+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:03.210790+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16093,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:04.016822+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:04.016822+0000","last_peered":"2024-09-10T20:31:04.016822+0000","last_clean":"2024-09-10T20:31:04.016822+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:04.016822+0000","last_undegraded":"2024-09-10T20:31:04.016822+0000","last_fullsized":"2024-09-10T20:31:04.016822+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12473,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211175+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:31:03.211175+0000","last_peered":"2024-09-10T20:31:03.211175+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:03.211175+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15986,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210690+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:03.210690+0000","last_peered":"2024-09-10T20:31:03.210690+0000","last_clean":"2024-09-10T20:31:03.210690+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:03.210690+0000","last_undegraded":"2024-09-10T20:31:03.210690+0000","last_fullsized":"2024-09-10T20:31:03.210690+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.215034+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:03.215034+0000","last_peered":"2024-09-10T20:31:03.215034+0000","last_clean":"2024-09-10T20:31:03.215034+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:03.215034+0000","last_undegraded":"2024-09-10T20:31:03.215034+0000","last_fullsized":"2024-09-10T20:31:03.215034+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12397,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:03.211805+0000","last_peered":"2024-09-10T20:31:03.211805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:03.211805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.212239+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:03.212239+0000","last_peered":"2024-09-10T20:31:03.212239+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:03.212239+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12027,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.215281+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:03.215281+0000","last_peered":"2024-09-10T20:31:03.215281+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:03.215281+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210499+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:03.210499+0000","last_peered":"2024-09-10T20:31:03.210499+0000","last_clean":"2024-09-10T20:31:03.210499+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:03.210499+0000","last_undegraded":"2024-09-10T20:31:03.210499+0000","last_fullsized":"2024-09-10T20:31:03.210499+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":14577,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:07.372655+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:07.372655+0000","last_peered":"2024-09-10T20:31:07.372655+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:07.372655+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16232,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210329+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:03.210329+0000","last_peered":"2024-09-10T20:31:03.210329+0000","last_clean":"2024-09-10T20:31:03.210329+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:03.210329+0000","last_undegraded":"2024-09-10T20:31:03.210329+0000","last_fullsized":"2024-09-10T20:31:03.210329+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703504,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668336,"statfs":{"total":96636764160,"available":95916376064,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573560,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798280,"statfs":{"total":96636764160,"available":96049438720,"internally_reserved":0,"allocated":430080,"data_stored":188432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699796,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672044,"statfs":{"total":96636764160,"available":95920173056,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:08.324 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:08.635 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:08.636 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:08.648 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-10T20:31:08.142407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676716,"kb_used_data":1560,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374810644,"statfs":{"total":386547056640,"available":383806099456,"internally_reserved":0,"allocated":1597440,"data_stored":630848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2462904,"num_objects":-302,"num_object_clones":0,"num_object_copies":-1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11766,"num_write_kb":-48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001494"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12418,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211994+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:03.211994+0000","last_peered":"2024-09-10T20:31:03.211994+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:03.211994+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12398,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210801+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:03.210801+0000","last_peered":"2024-09-10T20:31:03.210801+0000","last_clean":"2024-09-10T20:31:03.210801+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:03.210801+0000","last_undegraded":"2024-09-10T20:31:03.210801+0000","last_fullsized":"2024-09-10T20:31:03.210801+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12357,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.214988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:03.214988+0000","last_peered":"2024-09-10T20:31:03.214988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:03.214988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16589,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210258+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:03.210258+0000","last_peered":"2024-09-10T20:31:03.210258+0000","last_clean":"2024-09-10T20:31:03.210258+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:03.210258+0000","last_undegraded":"2024-09-10T20:31:03.210258+0000","last_fullsized":"2024-09-10T20:31:03.210258+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12501,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210284+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:03.210284+0000","last_peered":"2024-09-10T20:31:03.210284+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:03.210284+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12334,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210790+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:03.210790+0000","last_peered":"2024-09-10T20:31:03.210790+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:03.210790+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16093,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:04.016822+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:04.016822+0000","last_peered":"2024-09-10T20:31:04.016822+0000","last_clean":"2024-09-10T20:31:04.016822+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:04.016822+0000","last_undegraded":"2024-09-10T20:31:04.016822+0000","last_fullsized":"2024-09-10T20:31:04.016822+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12473,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211175+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:31:03.211175+0000","last_peered":"2024-09-10T20:31:03.211175+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:03.211175+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15986,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210690+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:03.210690+0000","last_peered":"2024-09-10T20:31:03.210690+0000","last_clean":"2024-09-10T20:31:03.210690+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:03.210690+0000","last_undegraded":"2024-09-10T20:31:03.210690+0000","last_fullsized":"2024-09-10T20:31:03.210690+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.215034+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:03.215034+0000","last_peered":"2024-09-10T20:31:03.215034+0000","last_clean":"2024-09-10T20:31:03.215034+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:03.215034+0000","last_undegraded":"2024-09-10T20:31:03.215034+0000","last_fullsized":"2024-09-10T20:31:03.215034+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12397,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:03.211805+0000","last_peered":"2024-09-10T20:31:03.211805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:03.211805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.212239+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:03.212239+0000","last_peered":"2024-09-10T20:31:03.212239+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:03.212239+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12027,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.215281+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:03.215281+0000","last_peered":"2024-09-10T20:31:03.215281+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:03.215281+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210499+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:03.210499+0000","last_peered":"2024-09-10T20:31:03.210499+0000","last_clean":"2024-09-10T20:31:03.210499+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:03.210499+0000","last_undegraded":"2024-09-10T20:31:03.210499+0000","last_fullsized":"2024-09-10T20:31:03.210499+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":14577,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:07.372655+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:07.372655+0000","last_peered":"2024-09-10T20:31:07.372655+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:07.372655+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16232,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210329+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:03.210329+0000","last_peered":"2024-09-10T20:31:03.210329+0000","last_clean":"2024-09-10T20:31:03.210329+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:03.210329+0000","last_undegraded":"2024-09-10T20:31:03.210329+0000","last_fullsized":"2024-09-10T20:31:03.210329+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703504,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668336,"statfs":{"total":96636764160,"available":95916376064,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573560,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798280,"statfs":{"total":96636764160,"available":96049438720,"internally_reserved":0,"allocated":430080,"data_stored":188432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699796,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672044,"statfs":{"total":96636764160,"available":95920173056,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:08.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:08.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12418, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.211994+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:03.211994+0000', 'last_peered': '2024-09-10T20:31:03.211994+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:03.211994+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:08.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12357, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.214988+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:03.214988+0000', 'last_peered': '2024-09-10T20:31:03.214988+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:03.214988+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12501, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.210284+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:03.210284+0000', 'last_peered': '2024-09-10T20:31:03.210284+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:03.210284+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 12334, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.210790+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:31:03.210790+0000', 'last_peered': '2024-09-10T20:31:03.210790+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:31:03.210790+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1857}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12291", 'reported_seq': 12473, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.211175+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:31:03.211175+0000', 'last_peered': '2024-09-10T20:31:03.211175+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:31:03.211175+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "32'2200", 'ondisk_log_start': "32'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12289, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12397, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.211805+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:03.211805+0000', 'last_peered': '2024-09-10T20:31:03.211805+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:03.211805+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12476, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.212239+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:03.212239+0000', 'last_peered': '2024-09-10T20:31:03.212239+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:03.212239+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:08.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:08.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12027, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.215281+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:31:03.215281+0000', 'last_peered': '2024-09-10T20:31:03.215281+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:03.215281+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:08.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:31:08.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12447", 'reported_seq': 14577, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:07.372655+0000', 'last_change': '2024-09-10T20:31:01.583331+0000', 'last_active': '2024-09-10T20:31:07.372655+0000', 'last_peered': '2024-09-10T20:31:07.372655+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:31:07.372655+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2400", 'ondisk_log_start': "32'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1011, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:08.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:09.038 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16059/52 objects degraded (30882.692%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":757,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13,"data_bytes":87568,"bytes_used":2740957184,"bytes_avail":383806099456,"bytes_total":386547056640,"degraded_objects":16059,"degraded_total":52,"degraded_ratio":308.82692307692309},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:09.051 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:31:09.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:09.378 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:09.378 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:09.391 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-10T20:31:08.142407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676716,"kb_used_data":1560,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374810644,"statfs":{"total":386547056640,"available":383806099456,"internally_reserved":0,"allocated":1597440,"data_stored":630848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2462904,"num_objects":-302,"num_object_clones":0,"num_object_copies":-1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11766,"num_write_kb":-48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001494"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12418,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211994+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:03.211994+0000","last_peered":"2024-09-10T20:31:03.211994+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:03.211994+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12398,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210801+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:03.210801+0000","last_peered":"2024-09-10T20:31:03.210801+0000","last_clean":"2024-09-10T20:31:03.210801+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:03.210801+0000","last_undegraded":"2024-09-10T20:31:03.210801+0000","last_fullsized":"2024-09-10T20:31:03.210801+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12357,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.214988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:03.214988+0000","last_peered":"2024-09-10T20:31:03.214988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:03.214988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16589,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210258+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:03.210258+0000","last_peered":"2024-09-10T20:31:03.210258+0000","last_clean":"2024-09-10T20:31:03.210258+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:03.210258+0000","last_undegraded":"2024-09-10T20:31:03.210258+0000","last_fullsized":"2024-09-10T20:31:03.210258+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12501,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210284+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:03.210284+0000","last_peered":"2024-09-10T20:31:03.210284+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:03.210284+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12334,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210790+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:03.210790+0000","last_peered":"2024-09-10T20:31:03.210790+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:03.210790+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16093,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:04.016822+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:04.016822+0000","last_peered":"2024-09-10T20:31:04.016822+0000","last_clean":"2024-09-10T20:31:04.016822+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:04.016822+0000","last_undegraded":"2024-09-10T20:31:04.016822+0000","last_fullsized":"2024-09-10T20:31:04.016822+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12473,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211175+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:31:03.211175+0000","last_peered":"2024-09-10T20:31:03.211175+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:03.211175+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15986,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210690+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:03.210690+0000","last_peered":"2024-09-10T20:31:03.210690+0000","last_clean":"2024-09-10T20:31:03.210690+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:03.210690+0000","last_undegraded":"2024-09-10T20:31:03.210690+0000","last_fullsized":"2024-09-10T20:31:03.210690+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.215034+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:03.215034+0000","last_peered":"2024-09-10T20:31:03.215034+0000","last_clean":"2024-09-10T20:31:03.215034+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:03.215034+0000","last_undegraded":"2024-09-10T20:31:03.215034+0000","last_fullsized":"2024-09-10T20:31:03.215034+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12397,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:03.211805+0000","last_peered":"2024-09-10T20:31:03.211805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:03.211805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.212239+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:03.212239+0000","last_peered":"2024-09-10T20:31:03.212239+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:03.212239+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12027,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.215281+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:03.215281+0000","last_peered":"2024-09-10T20:31:03.215281+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:03.215281+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210499+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:03.210499+0000","last_peered":"2024-09-10T20:31:03.210499+0000","last_clean":"2024-09-10T20:31:03.210499+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:03.210499+0000","last_undegraded":"2024-09-10T20:31:03.210499+0000","last_fullsized":"2024-09-10T20:31:03.210499+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":14577,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:07.372655+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:07.372655+0000","last_peered":"2024-09-10T20:31:07.372655+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:07.372655+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16232,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210329+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:03.210329+0000","last_peered":"2024-09-10T20:31:03.210329+0000","last_clean":"2024-09-10T20:31:03.210329+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:03.210329+0000","last_undegraded":"2024-09-10T20:31:03.210329+0000","last_fullsized":"2024-09-10T20:31:03.210329+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703504,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668336,"statfs":{"total":96636764160,"available":95916376064,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573560,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798280,"statfs":{"total":96636764160,"available":96049438720,"internally_reserved":0,"allocated":430080,"data_stored":188432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699796,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672044,"statfs":{"total":96636764160,"available":95920173056,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:12.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:12.710 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:12.711 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:12.723 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-10T20:31:12.143232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685024,"kb_used_data":1560,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374802336,"statfs":{"total":386547056640,"available":383797592064,"internally_reserved":0,"allocated":1597440,"data_stored":630848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2747778614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12066,"num_write_kb":-48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.934337"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12418,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211994+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:03.211994+0000","last_peered":"2024-09-10T20:31:03.211994+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:03.211994+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12398,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210801+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:03.210801+0000","last_peered":"2024-09-10T20:31:03.210801+0000","last_clean":"2024-09-10T20:31:03.210801+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:03.210801+0000","last_undegraded":"2024-09-10T20:31:03.210801+0000","last_fullsized":"2024-09-10T20:31:03.210801+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12357,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.214988+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:03.214988+0000","last_peered":"2024-09-10T20:31:03.214988+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:03.214988+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16589,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210258+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:03.210258+0000","last_peered":"2024-09-10T20:31:03.210258+0000","last_clean":"2024-09-10T20:31:03.210258+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:03.210258+0000","last_undegraded":"2024-09-10T20:31:03.210258+0000","last_fullsized":"2024-09-10T20:31:03.210258+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12501,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210284+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:03.210284+0000","last_peered":"2024-09-10T20:31:03.210284+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:03.210284+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12334,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210790+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:03.210790+0000","last_peered":"2024-09-10T20:31:03.210790+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:03.210790+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16093,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:04.016822+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:04.016822+0000","last_peered":"2024-09-10T20:31:04.016822+0000","last_clean":"2024-09-10T20:31:04.016822+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:04.016822+0000","last_undegraded":"2024-09-10T20:31:04.016822+0000","last_fullsized":"2024-09-10T20:31:04.016822+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12473,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211175+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:31:03.211175+0000","last_peered":"2024-09-10T20:31:03.211175+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:03.211175+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15986,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210690+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:03.210690+0000","last_peered":"2024-09-10T20:31:03.210690+0000","last_clean":"2024-09-10T20:31:03.210690+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:03.210690+0000","last_undegraded":"2024-09-10T20:31:03.210690+0000","last_fullsized":"2024-09-10T20:31:03.210690+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.215034+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:03.215034+0000","last_peered":"2024-09-10T20:31:03.215034+0000","last_clean":"2024-09-10T20:31:03.215034+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:03.215034+0000","last_undegraded":"2024-09-10T20:31:03.215034+0000","last_fullsized":"2024-09-10T20:31:03.215034+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12397,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:03.211805+0000","last_peered":"2024-09-10T20:31:03.211805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:03.211805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.212239+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:03.212239+0000","last_peered":"2024-09-10T20:31:03.212239+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:03.212239+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12027,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.215281+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:03.215281+0000","last_peered":"2024-09-10T20:31:03.215281+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:03.215281+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210499+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:03.210499+0000","last_peered":"2024-09-10T20:31:03.210499+0000","last_clean":"2024-09-10T20:31:03.210499+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:03.210499+0000","last_undegraded":"2024-09-10T20:31:03.210499+0000","last_fullsized":"2024-09-10T20:31:03.210499+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":14577,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:07.372655+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:07.372655+0000","last_peered":"2024-09-10T20:31:07.372655+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:07.372655+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16232,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210329+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:03.210329+0000","last_peered":"2024-09-10T20:31:03.210329+0000","last_clean":"2024-09-10T20:31:03.210329+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:03.210329+0000","last_undegraded":"2024-09-10T20:31:03.210329+0000","last_fullsized":"2024-09-10T20:31:03.210329+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703504,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668336,"statfs":{"total":96636764160,"available":95916376064,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581876,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789964,"statfs":{"total":96636764160,"available":96040923136,"internally_reserved":0,"allocated":430080,"data_stored":188432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699788,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672052,"statfs":{"total":96636764160,"available":95920181248,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:12.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:13.034 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:13.048 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-10T20:31:12.728843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685032,"kb_used_data":1568,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374802328,"statfs":{"total":386547056640,"available":383797583872,"internally_reserved":0,"allocated":1605632,"data_stored":634212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2747778614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12066,"num_write_kb":-48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.587233"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12418,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211994+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:03.211994+0000","last_peered":"2024-09-10T20:31:03.211994+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:03.211994+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12398,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210801+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:03.210801+0000","last_peered":"2024-09-10T20:31:03.210801+0000","last_clean":"2024-09-10T20:31:03.210801+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:03.210801+0000","last_undegraded":"2024-09-10T20:31:03.210801+0000","last_fullsized":"2024-09-10T20:31:03.210801+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12359,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731801+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:11.731801+0000","last_peered":"2024-09-10T20:31:11.731801+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:11.731801+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16589,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210258+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:03.210258+0000","last_peered":"2024-09-10T20:31:03.210258+0000","last_clean":"2024-09-10T20:31:03.210258+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:03.210258+0000","last_undegraded":"2024-09-10T20:31:03.210258+0000","last_fullsized":"2024-09-10T20:31:03.210258+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12501,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210284+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:03.210284+0000","last_peered":"2024-09-10T20:31:03.210284+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:03.210284+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12334,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210790+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:03.210790+0000","last_peered":"2024-09-10T20:31:03.210790+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:03.210790+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16093,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:04.016822+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:04.016822+0000","last_peered":"2024-09-10T20:31:04.016822+0000","last_clean":"2024-09-10T20:31:04.016822+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:04.016822+0000","last_undegraded":"2024-09-10T20:31:04.016822+0000","last_fullsized":"2024-09-10T20:31:04.016822+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12473,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211175+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:31:03.211175+0000","last_peered":"2024-09-10T20:31:03.211175+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:03.211175+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15986,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210690+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:03.210690+0000","last_peered":"2024-09-10T20:31:03.210690+0000","last_clean":"2024-09-10T20:31:03.210690+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:03.210690+0000","last_undegraded":"2024-09-10T20:31:03.210690+0000","last_fullsized":"2024-09-10T20:31:03.210690+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16128,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-10T20:31:11.732790+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:11.732790+0000","last_peered":"2024-09-10T20:31:11.732790+0000","last_clean":"2024-09-10T20:31:11.732790+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:11.732790+0000","last_undegraded":"2024-09-10T20:31:11.732790+0000","last_fullsized":"2024-09-10T20:31:11.732790+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12397,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:03.211805+0000","last_peered":"2024-09-10T20:31:03.211805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:03.211805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.212239+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:03.212239+0000","last_peered":"2024-09-10T20:31:03.212239+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:03.212239+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12029,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731641+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:11.731641+0000","last_peered":"2024-09-10T20:31:11.731641+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:11.731641+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210499+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:03.210499+0000","last_peered":"2024-09-10T20:31:03.210499+0000","last_clean":"2024-09-10T20:31:03.210499+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:03.210499+0000","last_undegraded":"2024-09-10T20:31:03.210499+0000","last_fullsized":"2024-09-10T20:31:03.210499+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":14577,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:07.372655+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:07.372655+0000","last_peered":"2024-09-10T20:31:07.372655+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:07.372655+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16232,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210329+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:03.210329+0000","last_peered":"2024-09-10T20:31:03.210329+0000","last_clean":"2024-09-10T20:31:03.210329+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:03.210329+0000","last_undegraded":"2024-09-10T20:31:03.210329+0000","last_fullsized":"2024-09-10T20:31:03.210329+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703512,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668328,"statfs":{"total":96636764160,"available":95916367872,"internally_reserved":0,"allocated":397312,"data_stored":150836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581876,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789964,"statfs":{"total":96636764160,"available":96040923136,"internally_reserved":0,"allocated":430080,"data_stored":188432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699788,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672052,"statfs":{"total":96636764160,"available":95920181248,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12418, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.211994+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:03.211994+0000', 'last_peered': '2024-09-10T20:31:03.211994+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:03.211994+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12359, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:11.731801+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:11.731801+0000', 'last_peered': '2024-09-10T20:31:11.731801+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:11.731801+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12501, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.210284+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:03.210284+0000', 'last_peered': '2024-09-10T20:31:03.210284+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:03.210284+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:13.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 12334, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.210790+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:31:03.210790+0000', 'last_peered': '2024-09-10T20:31:03.210790+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:31:03.210790+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1857}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12291", 'reported_seq': 12473, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.211175+0000', 'last_change': '2024-09-10T20:20:16.156515+0000', 'last_active': '2024-09-10T20:31:03.211175+0000', 'last_peered': '2024-09-10T20:31:03.211175+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:31:03.211175+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "32'2200", 'ondisk_log_start': "32'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12289, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12397, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.211805+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:03.211805+0000', 'last_peered': '2024-09-10T20:31:03.211805+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:03.211805+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12476, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:03.212239+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:03.212239+0000', 'last_peered': '2024-09-10T20:31:03.212239+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:03.212239+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12029, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:11.731641+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:31:11.731641+0000', 'last_peered': '2024-09-10T20:31:11.731641+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:11.731641+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-10T20:31:13.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "44'12447", 'reported_seq': 14577, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:07.372655+0000', 'last_change': '2024-09-10T20:31:01.583331+0000', 'last_active': '2024-09-10T20:31:07.372655+0000', 'last_peered': '2024-09-10T20:31:07.372655+0000', 'last_clean': '2024-09-10T20:19:06.311861+0000', 'last_became_active': '2024-09-10T20:20:16.112549+0000', 'last_became_peered': '2024-09-10T20:20:16.112549+0000', 'last_unstale': '2024-09-10T20:31:07.372655+0000', 'last_undegraded': '2024-09-10T20:20:15.485369+0000', 'last_fullsized': '2024-09-10T20:20:15.458132+0000', 'mapping_epoch': 31, 'log_start': "32'2400", 'ondisk_log_start': "32'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1011, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:13.051 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:13.487 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16059/52 objects degraded (30882.692%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":762,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13,"data_bytes":87568,"bytes_used":2749464576,"bytes_avail":383797592064,"bytes_total":386547056640,"degraded_objects":16059,"degraded_total":52,"degraded_ratio":308.82692307692309},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:13.501 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:31:13.501 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:13.808 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:13.809 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:13.825 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-10T20:31:12.728843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685032,"kb_used_data":1568,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374802328,"statfs":{"total":386547056640,"available":383797583872,"internally_reserved":0,"allocated":1605632,"data_stored":634212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2747778614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12066,"num_write_kb":-48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.587233"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12418,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211994+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:03.211994+0000","last_peered":"2024-09-10T20:31:03.211994+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:03.211994+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12398,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210801+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:03.210801+0000","last_peered":"2024-09-10T20:31:03.210801+0000","last_clean":"2024-09-10T20:31:03.210801+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:03.210801+0000","last_undegraded":"2024-09-10T20:31:03.210801+0000","last_fullsized":"2024-09-10T20:31:03.210801+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12359,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731801+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:11.731801+0000","last_peered":"2024-09-10T20:31:11.731801+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:11.731801+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16589,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210258+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:03.210258+0000","last_peered":"2024-09-10T20:31:03.210258+0000","last_clean":"2024-09-10T20:31:03.210258+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:03.210258+0000","last_undegraded":"2024-09-10T20:31:03.210258+0000","last_fullsized":"2024-09-10T20:31:03.210258+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12501,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210284+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:03.210284+0000","last_peered":"2024-09-10T20:31:03.210284+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:03.210284+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12334,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.210790+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:03.210790+0000","last_peered":"2024-09-10T20:31:03.210790+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:03.210790+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16093,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:04.016822+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:04.016822+0000","last_peered":"2024-09-10T20:31:04.016822+0000","last_clean":"2024-09-10T20:31:04.016822+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:04.016822+0000","last_undegraded":"2024-09-10T20:31:04.016822+0000","last_fullsized":"2024-09-10T20:31:04.016822+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12473,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211175+0000","last_change":"2024-09-10T20:20:16.156515+0000","last_active":"2024-09-10T20:31:03.211175+0000","last_peered":"2024-09-10T20:31:03.211175+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:03.211175+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15986,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210690+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:03.210690+0000","last_peered":"2024-09-10T20:31:03.210690+0000","last_clean":"2024-09-10T20:31:03.210690+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:03.210690+0000","last_undegraded":"2024-09-10T20:31:03.210690+0000","last_fullsized":"2024-09-10T20:31:03.210690+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16128,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-10T20:31:11.732790+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:11.732790+0000","last_peered":"2024-09-10T20:31:11.732790+0000","last_clean":"2024-09-10T20:31:11.732790+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:11.732790+0000","last_undegraded":"2024-09-10T20:31:11.732790+0000","last_fullsized":"2024-09-10T20:31:11.732790+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12397,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.211805+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:03.211805+0000","last_peered":"2024-09-10T20:31:03.211805+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:03.211805+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:03.212239+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:03.212239+0000","last_peered":"2024-09-10T20:31:03.212239+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:03.212239+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12029,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731641+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:11.731641+0000","last_peered":"2024-09-10T20:31:11.731641+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:11.731641+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210499+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:03.210499+0000","last_peered":"2024-09-10T20:31:03.210499+0000","last_clean":"2024-09-10T20:31:03.210499+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:03.210499+0000","last_undegraded":"2024-09-10T20:31:03.210499+0000","last_fullsized":"2024-09-10T20:31:03.210499+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":14577,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:07.372655+0000","last_change":"2024-09-10T20:31:01.583331+0000","last_active":"2024-09-10T20:31:07.372655+0000","last_peered":"2024-09-10T20:31:07.372655+0000","last_clean":"2024-09-10T20:19:06.311861+0000","last_became_active":"2024-09-10T20:20:16.112549+0000","last_became_peered":"2024-09-10T20:20:16.112549+0000","last_unstale":"2024-09-10T20:31:07.372655+0000","last_undegraded":"2024-09-10T20:20:15.485369+0000","last_fullsized":"2024-09-10T20:20:15.458132+0000","mapping_epoch":31,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16232,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-10T20:31:03.210329+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:03.210329+0000","last_peered":"2024-09-10T20:31:03.210329+0000","last_clean":"2024-09-10T20:31:03.210329+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:03.210329+0000","last_undegraded":"2024-09-10T20:31:03.210329+0000","last_fullsized":"2024-09-10T20:31:03.210329+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87568,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142089,"num_write_kb":482223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703512,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668328,"statfs":{"total":96636764160,"available":95916367872,"internally_reserved":0,"allocated":397312,"data_stored":150836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581876,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789964,"statfs":{"total":96636764160,"available":96040923136,"internally_reserved":0,"allocated":430080,"data_stored":188432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699788,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672052,"statfs":{"total":96636764160,"available":95920181248,"internally_reserved":0,"allocated":389120,"data_stored":147472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:16.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:17.146 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:17.147 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:17.158 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-10T20:31:16.143857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676772,"kb_used_data":1608,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374810588,"statfs":{"total":386547056640,"available":383806042112,"internally_reserved":0,"allocated":1646592,"data_stored":646141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001449"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12422,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732630+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:12.732630+0000","last_peered":"2024-09-10T20:31:12.732630+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:12.732630+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735828+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:12.735828+0000","last_peered":"2024-09-10T20:31:12.735828+0000","last_clean":"2024-09-10T20:31:12.735828+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:12.735828+0000","last_undegraded":"2024-09-10T20:31:12.735828+0000","last_fullsized":"2024-09-10T20:31:12.735828+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12359,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731801+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:11.731801+0000","last_peered":"2024-09-10T20:31:11.731801+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:11.731801+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16593,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735033+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:12.735033+0000","last_peered":"2024-09-10T20:31:12.735033+0000","last_clean":"2024-09-10T20:31:12.735033+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:12.735033+0000","last_undegraded":"2024-09-10T20:31:12.735033+0000","last_fullsized":"2024-09-10T20:31:12.735033+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731714+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:12.731714+0000","last_peered":"2024-09-10T20:31:12.731714+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:12.731714+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12338,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731668+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:12.731668+0000","last_peered":"2024-09-10T20:31:12.731668+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:12.731668+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16097,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731237+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:12.731237+0000","last_peered":"2024-09-10T20:31:12.731237+0000","last_clean":"2024-09-10T20:31:12.731237+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:12.731237+0000","last_undegraded":"2024-09-10T20:31:12.731237+0000","last_fullsized":"2024-09-10T20:31:12.731237+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":13190,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:14.223807+0000","last_change":"2024-09-10T20:31:11.673002+0000","last_active":"2024-09-10T20:31:14.223807+0000","last_peered":"2024-09-10T20:31:14.223807+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:14.223807+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":401,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731530+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:12.731530+0000","last_peered":"2024-09-10T20:31:12.731530+0000","last_clean":"2024-09-10T20:31:12.731530+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:12.731530+0000","last_undegraded":"2024-09-10T20:31:12.731530+0000","last_fullsized":"2024-09-10T20:31:12.731530+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16128,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-10T20:31:11.732790+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:11.732790+0000","last_peered":"2024-09-10T20:31:11.732790+0000","last_clean":"2024-09-10T20:31:11.732790+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:11.732790+0000","last_undegraded":"2024-09-10T20:31:11.732790+0000","last_fullsized":"2024-09-10T20:31:11.732790+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12401,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732241+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:12.732241+0000","last_peered":"2024-09-10T20:31:12.732241+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:12.732241+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12480,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.733198+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:12.733198+0000","last_peered":"2024-09-10T20:31:12.733198+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:12.733198+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12029,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731641+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:11.731641+0000","last_peered":"2024-09-10T20:31:11.731641+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:11.731641+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.736102+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:12.736102+0000","last_peered":"2024-09-10T20:31:12.736102+0000","last_clean":"2024-09-10T20:31:12.736102+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:12.736102+0000","last_undegraded":"2024-09-10T20:31:12.736102+0000","last_fullsized":"2024-09-10T20:31:12.736102+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12685,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:13.134763+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:13.134763+0000","last_peered":"2024-09-10T20:31:13.134763+0000","last_clean":"2024-09-10T20:31:13.134763+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:13.134763+0000","last_undegraded":"2024-09-10T20:31:13.134763+0000","last_fullsized":"2024-09-10T20:31:13.134763+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16236,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.734331+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:12.734331+0000","last_peered":"2024-09-10T20:31:12.734331+0000","last_clean":"2024-09-10T20:31:12.734331+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:12.734331+0000","last_undegraded":"2024-09-10T20:31:12.734331+0000","last_fullsized":"2024-09-10T20:31:12.734331+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703512,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668328,"statfs":{"total":96636764160,"available":95916367872,"internally_reserved":0,"allocated":397312,"data_stored":150836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573572,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798268,"statfs":{"total":96636764160,"available":96049426432,"internally_reserved":0,"allocated":438272,"data_stored":186947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699812,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672028,"statfs":{"total":96636764160,"available":95920156672,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:17.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:17.462 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:17.463 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:17.475 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-10T20:31:16.143857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676772,"kb_used_data":1608,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374810588,"statfs":{"total":386547056640,"available":383806042112,"internally_reserved":0,"allocated":1646592,"data_stored":646141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001449"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12422,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732630+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:12.732630+0000","last_peered":"2024-09-10T20:31:12.732630+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:12.732630+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735828+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:12.735828+0000","last_peered":"2024-09-10T20:31:12.735828+0000","last_clean":"2024-09-10T20:31:12.735828+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:12.735828+0000","last_undegraded":"2024-09-10T20:31:12.735828+0000","last_fullsized":"2024-09-10T20:31:12.735828+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12359,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731801+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:11.731801+0000","last_peered":"2024-09-10T20:31:11.731801+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:11.731801+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16593,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735033+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:12.735033+0000","last_peered":"2024-09-10T20:31:12.735033+0000","last_clean":"2024-09-10T20:31:12.735033+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:12.735033+0000","last_undegraded":"2024-09-10T20:31:12.735033+0000","last_fullsized":"2024-09-10T20:31:12.735033+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731714+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:12.731714+0000","last_peered":"2024-09-10T20:31:12.731714+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:12.731714+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12338,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731668+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:12.731668+0000","last_peered":"2024-09-10T20:31:12.731668+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:12.731668+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16097,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731237+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:12.731237+0000","last_peered":"2024-09-10T20:31:12.731237+0000","last_clean":"2024-09-10T20:31:12.731237+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:12.731237+0000","last_undegraded":"2024-09-10T20:31:12.731237+0000","last_fullsized":"2024-09-10T20:31:12.731237+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":13190,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:14.223807+0000","last_change":"2024-09-10T20:31:11.673002+0000","last_active":"2024-09-10T20:31:14.223807+0000","last_peered":"2024-09-10T20:31:14.223807+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:14.223807+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":401,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731530+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:12.731530+0000","last_peered":"2024-09-10T20:31:12.731530+0000","last_clean":"2024-09-10T20:31:12.731530+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:12.731530+0000","last_undegraded":"2024-09-10T20:31:12.731530+0000","last_fullsized":"2024-09-10T20:31:12.731530+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16128,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-10T20:31:11.732790+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:11.732790+0000","last_peered":"2024-09-10T20:31:11.732790+0000","last_clean":"2024-09-10T20:31:11.732790+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:11.732790+0000","last_undegraded":"2024-09-10T20:31:11.732790+0000","last_fullsized":"2024-09-10T20:31:11.732790+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12401,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732241+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:12.732241+0000","last_peered":"2024-09-10T20:31:12.732241+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:12.732241+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12480,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.733198+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:12.733198+0000","last_peered":"2024-09-10T20:31:12.733198+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:12.733198+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12029,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:11.731641+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:11.731641+0000","last_peered":"2024-09-10T20:31:11.731641+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:11.731641+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.736102+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:12.736102+0000","last_peered":"2024-09-10T20:31:12.736102+0000","last_clean":"2024-09-10T20:31:12.736102+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:12.736102+0000","last_undegraded":"2024-09-10T20:31:12.736102+0000","last_fullsized":"2024-09-10T20:31:12.736102+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12685,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:13.134763+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:13.134763+0000","last_peered":"2024-09-10T20:31:13.134763+0000","last_clean":"2024-09-10T20:31:13.134763+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:13.134763+0000","last_undegraded":"2024-09-10T20:31:13.134763+0000","last_fullsized":"2024-09-10T20:31:13.134763+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16236,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.734331+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:12.734331+0000","last_peered":"2024-09-10T20:31:12.734331+0000","last_clean":"2024-09-10T20:31:12.734331+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:12.734331+0000","last_undegraded":"2024-09-10T20:31:12.734331+0000","last_fullsized":"2024-09-10T20:31:12.734331+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703512,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668328,"statfs":{"total":96636764160,"available":95916367872,"internally_reserved":0,"allocated":397312,"data_stored":150836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573572,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798268,"statfs":{"total":96636764160,"available":96049426432,"internally_reserved":0,"allocated":438272,"data_stored":186947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699812,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672028,"statfs":{"total":96636764160,"available":95920156672,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:17.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:17.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12422, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732630+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:12.732630+0000', 'last_peered': '2024-09-10T20:31:12.732630+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:12.732630+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12359, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:11.731801+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:11.731801+0000', 'last_peered': '2024-09-10T20:31:11.731801+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:11.731801+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12505, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.731714+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:12.731714+0000', 'last_peered': '2024-09-10T20:31:12.731714+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:12.731714+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 12338, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.731668+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:31:12.731668+0000', 'last_peered': '2024-09-10T20:31:12.731668+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:31:12.731668+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1857}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12291", 'reported_seq': 13190, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:14.223807+0000', 'last_change': '2024-09-10T20:31:11.673002+0000', 'last_active': '2024-09-10T20:31:14.223807+0000', 'last_peered': '2024-09-10T20:31:14.223807+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:31:14.223807+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "32'2200", 'ondisk_log_start': "32'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1646, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12289, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 401, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1644}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:17.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12401, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732241+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:12.732241+0000', 'last_peered': '2024-09-10T20:31:12.732241+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:12.732241+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:17.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:17.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12480, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.733198+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:12.733198+0000', 'last_peered': '2024-09-10T20:31:12.733198+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:12.733198+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:17.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:17.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12029, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:11.731641+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:31:11.731641+0000', 'last_peered': '2024-09-10T20:31:11.731641+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:11.731641+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:17.478 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:17.882 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:17.883 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14748/44 objects degraded (33518.182%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":766,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11,"data_bytes":74096,"bytes_used":2741014528,"bytes_avail":383806042112,"bytes_total":386547056640,"degraded_objects":14748,"degraded_total":44,"degraded_ratio":335.18181818181819},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:17.896 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:31:17.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:18.239 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:18.251 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-10T20:31:18.144218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676780,"kb_used_data":1616,"kb_used_omap":14,"kb_used_meta":2675057,"kb_avail":374810580,"statfs":{"total":386547056640,"available":383806033920,"internally_reserved":0,"allocated":1654784,"data_stored":649484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2739258934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001345"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12422,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732630+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:12.732630+0000","last_peered":"2024-09-10T20:31:12.732630+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:12.732630+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735828+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:12.735828+0000","last_peered":"2024-09-10T20:31:12.735828+0000","last_clean":"2024-09-10T20:31:12.735828+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:12.735828+0000","last_undegraded":"2024-09-10T20:31:12.735828+0000","last_fullsized":"2024-09-10T20:31:12.735828+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12361,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732017+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:12.732017+0000","last_peered":"2024-09-10T20:31:12.732017+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:12.732017+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16593,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735033+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:12.735033+0000","last_peered":"2024-09-10T20:31:12.735033+0000","last_clean":"2024-09-10T20:31:12.735033+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:12.735033+0000","last_undegraded":"2024-09-10T20:31:12.735033+0000","last_fullsized":"2024-09-10T20:31:12.735033+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731714+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:12.731714+0000","last_peered":"2024-09-10T20:31:12.731714+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:12.731714+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12338,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731668+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:12.731668+0000","last_peered":"2024-09-10T20:31:12.731668+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:12.731668+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16097,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731237+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:12.731237+0000","last_peered":"2024-09-10T20:31:12.731237+0000","last_clean":"2024-09-10T20:31:12.731237+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:12.731237+0000","last_undegraded":"2024-09-10T20:31:12.731237+0000","last_fullsized":"2024-09-10T20:31:12.731237+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":13190,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:14.223807+0000","last_change":"2024-09-10T20:31:11.673002+0000","last_active":"2024-09-10T20:31:14.223807+0000","last_peered":"2024-09-10T20:31:14.223807+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:14.223807+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":401,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731530+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:12.731530+0000","last_peered":"2024-09-10T20:31:12.731530+0000","last_clean":"2024-09-10T20:31:12.731530+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:12.731530+0000","last_undegraded":"2024-09-10T20:31:12.731530+0000","last_fullsized":"2024-09-10T20:31:12.731530+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16130,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.732926+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:12.732926+0000","last_peered":"2024-09-10T20:31:12.732926+0000","last_clean":"2024-09-10T20:31:12.732926+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:12.732926+0000","last_undegraded":"2024-09-10T20:31:12.732926+0000","last_fullsized":"2024-09-10T20:31:12.732926+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12401,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732241+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:12.732241+0000","last_peered":"2024-09-10T20:31:12.732241+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:12.732241+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12480,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.733198+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:12.733198+0000","last_peered":"2024-09-10T20:31:12.733198+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:12.733198+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12031,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732645+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:12.732645+0000","last_peered":"2024-09-10T20:31:12.732645+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:12.732645+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.736102+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:12.736102+0000","last_peered":"2024-09-10T20:31:12.736102+0000","last_clean":"2024-09-10T20:31:12.736102+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:12.736102+0000","last_undegraded":"2024-09-10T20:31:12.736102+0000","last_fullsized":"2024-09-10T20:31:12.736102+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12685,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:13.134763+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:13.134763+0000","last_peered":"2024-09-10T20:31:13.134763+0000","last_clean":"2024-09-10T20:31:13.134763+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:13.134763+0000","last_undegraded":"2024-09-10T20:31:13.134763+0000","last_fullsized":"2024-09-10T20:31:13.134763+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16236,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.734331+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:12.734331+0000","last_peered":"2024-09-10T20:31:12.734331+0000","last_clean":"2024-09-10T20:31:12.734331+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:12.734331+0000","last_undegraded":"2024-09-10T20:31:12.734331+0000","last_fullsized":"2024-09-10T20:31:12.734331+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703520,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668320,"statfs":{"total":96636764160,"available":95916359680,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573572,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798268,"statfs":{"total":96636764160,"available":96049426432,"internally_reserved":0,"allocated":438272,"data_stored":186947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699812,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672028,"statfs":{"total":96636764160,"available":95920156672,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:21.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:21.557 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:21.557 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:21.569 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-10T20:31:20.144537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680932,"kb_used_data":1616,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806428,"statfs":{"total":386547056640,"available":383801782272,"internally_reserved":0,"allocated":1654784,"data_stored":649484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.417399"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12422,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732630+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:12.732630+0000","last_peered":"2024-09-10T20:31:12.732630+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:12.732630+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735828+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:12.735828+0000","last_peered":"2024-09-10T20:31:12.735828+0000","last_clean":"2024-09-10T20:31:12.735828+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:12.735828+0000","last_undegraded":"2024-09-10T20:31:12.735828+0000","last_fullsized":"2024-09-10T20:31:12.735828+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12361,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732017+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:12.732017+0000","last_peered":"2024-09-10T20:31:12.732017+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:12.732017+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16594,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735089+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:12.735089+0000","last_peered":"2024-09-10T20:31:12.735089+0000","last_clean":"2024-09-10T20:31:12.735089+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:12.735089+0000","last_undegraded":"2024-09-10T20:31:12.735089+0000","last_fullsized":"2024-09-10T20:31:12.735089+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731714+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:12.731714+0000","last_peered":"2024-09-10T20:31:12.731714+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:12.731714+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12338,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731668+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:12.731668+0000","last_peered":"2024-09-10T20:31:12.731668+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:12.731668+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16097,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731237+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:12.731237+0000","last_peered":"2024-09-10T20:31:12.731237+0000","last_clean":"2024-09-10T20:31:12.731237+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:12.731237+0000","last_undegraded":"2024-09-10T20:31:12.731237+0000","last_fullsized":"2024-09-10T20:31:12.731237+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":14988,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:19.223485+0000","last_change":"2024-09-10T20:31:11.673002+0000","last_active":"2024-09-10T20:31:19.223485+0000","last_peered":"2024-09-10T20:31:19.223485+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:19.223485+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1300,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731530+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:12.731530+0000","last_peered":"2024-09-10T20:31:12.731530+0000","last_clean":"2024-09-10T20:31:12.731530+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:12.731530+0000","last_undegraded":"2024-09-10T20:31:12.731530+0000","last_fullsized":"2024-09-10T20:31:12.731530+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16130,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.732926+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:12.732926+0000","last_peered":"2024-09-10T20:31:12.732926+0000","last_clean":"2024-09-10T20:31:12.732926+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:12.732926+0000","last_undegraded":"2024-09-10T20:31:12.732926+0000","last_fullsized":"2024-09-10T20:31:12.732926+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12401,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732241+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:12.732241+0000","last_peered":"2024-09-10T20:31:12.732241+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:12.732241+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12480,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.733198+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:12.733198+0000","last_peered":"2024-09-10T20:31:12.733198+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:12.733198+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12031,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732645+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:12.732645+0000","last_peered":"2024-09-10T20:31:12.732645+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:12.732645+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.736102+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:12.736102+0000","last_peered":"2024-09-10T20:31:12.736102+0000","last_clean":"2024-09-10T20:31:12.736102+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:12.736102+0000","last_undegraded":"2024-09-10T20:31:12.736102+0000","last_fullsized":"2024-09-10T20:31:12.736102+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12685,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:13.134763+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:13.134763+0000","last_peered":"2024-09-10T20:31:13.134763+0000","last_clean":"2024-09-10T20:31:13.134763+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:13.134763+0000","last_undegraded":"2024-09-10T20:31:13.134763+0000","last_fullsized":"2024-09-10T20:31:13.134763+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16236,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.734331+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:12.734331+0000","last_peered":"2024-09-10T20:31:12.734331+0000","last_clean":"2024-09-10T20:31:12.734331+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:12.734331+0000","last_undegraded":"2024-09-10T20:31:12.734331+0000","last_fullsized":"2024-09-10T20:31:12.734331+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703520,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668320,"statfs":{"total":96636764160,"available":95916359680,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577724,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794116,"statfs":{"total":96636764160,"available":96045174784,"internally_reserved":0,"allocated":438272,"data_stored":186947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699812,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672028,"statfs":{"total":96636764160,"available":95920156672,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:21.570 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:21.907 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:21.919 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-10T20:31:20.144537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680932,"kb_used_data":1616,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806428,"statfs":{"total":386547056640,"available":383801782272,"internally_reserved":0,"allocated":1654784,"data_stored":649484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.417399"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12422,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732630+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:12.732630+0000","last_peered":"2024-09-10T20:31:12.732630+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:12.732630+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735828+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:12.735828+0000","last_peered":"2024-09-10T20:31:12.735828+0000","last_clean":"2024-09-10T20:31:12.735828+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:12.735828+0000","last_undegraded":"2024-09-10T20:31:12.735828+0000","last_fullsized":"2024-09-10T20:31:12.735828+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12361,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732017+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:12.732017+0000","last_peered":"2024-09-10T20:31:12.732017+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:12.732017+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16594,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735089+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:12.735089+0000","last_peered":"2024-09-10T20:31:12.735089+0000","last_clean":"2024-09-10T20:31:12.735089+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:12.735089+0000","last_undegraded":"2024-09-10T20:31:12.735089+0000","last_fullsized":"2024-09-10T20:31:12.735089+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731714+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:12.731714+0000","last_peered":"2024-09-10T20:31:12.731714+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:12.731714+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12338,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731668+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:12.731668+0000","last_peered":"2024-09-10T20:31:12.731668+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:12.731668+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16097,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731237+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:12.731237+0000","last_peered":"2024-09-10T20:31:12.731237+0000","last_clean":"2024-09-10T20:31:12.731237+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:12.731237+0000","last_undegraded":"2024-09-10T20:31:12.731237+0000","last_fullsized":"2024-09-10T20:31:12.731237+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":14988,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:19.223485+0000","last_change":"2024-09-10T20:31:11.673002+0000","last_active":"2024-09-10T20:31:19.223485+0000","last_peered":"2024-09-10T20:31:19.223485+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:19.223485+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1300,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731530+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:12.731530+0000","last_peered":"2024-09-10T20:31:12.731530+0000","last_clean":"2024-09-10T20:31:12.731530+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:12.731530+0000","last_undegraded":"2024-09-10T20:31:12.731530+0000","last_fullsized":"2024-09-10T20:31:12.731530+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16130,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.732926+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:12.732926+0000","last_peered":"2024-09-10T20:31:12.732926+0000","last_clean":"2024-09-10T20:31:12.732926+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:12.732926+0000","last_undegraded":"2024-09-10T20:31:12.732926+0000","last_fullsized":"2024-09-10T20:31:12.732926+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12401,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732241+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:12.732241+0000","last_peered":"2024-09-10T20:31:12.732241+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:12.732241+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12480,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.733198+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:12.733198+0000","last_peered":"2024-09-10T20:31:12.733198+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:12.733198+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12031,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732645+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:12.732645+0000","last_peered":"2024-09-10T20:31:12.732645+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:12.732645+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.736102+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:12.736102+0000","last_peered":"2024-09-10T20:31:12.736102+0000","last_clean":"2024-09-10T20:31:12.736102+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:12.736102+0000","last_undegraded":"2024-09-10T20:31:12.736102+0000","last_fullsized":"2024-09-10T20:31:12.736102+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12685,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:13.134763+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:13.134763+0000","last_peered":"2024-09-10T20:31:13.134763+0000","last_clean":"2024-09-10T20:31:13.134763+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:13.134763+0000","last_undegraded":"2024-09-10T20:31:13.134763+0000","last_fullsized":"2024-09-10T20:31:13.134763+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16236,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.734331+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:12.734331+0000","last_peered":"2024-09-10T20:31:12.734331+0000","last_clean":"2024-09-10T20:31:12.734331+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:12.734331+0000","last_undegraded":"2024-09-10T20:31:12.734331+0000","last_fullsized":"2024-09-10T20:31:12.734331+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703520,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668320,"statfs":{"total":96636764160,"available":95916359680,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577724,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794116,"statfs":{"total":96636764160,"available":96045174784,"internally_reserved":0,"allocated":438272,"data_stored":186947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699812,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672028,"statfs":{"total":96636764160,"available":95920156672,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:21.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:21.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12422, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732630+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:12.732630+0000', 'last_peered': '2024-09-10T20:31:12.732630+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:12.732630+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:21.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:21.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12361, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732017+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:12.732017+0000', 'last_peered': '2024-09-10T20:31:12.732017+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:12.732017+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:21.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:21.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12505, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.731714+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:12.731714+0000', 'last_peered': '2024-09-10T20:31:12.731714+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:12.731714+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:21.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:21.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 12338, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.731668+0000', 'last_change': '2024-09-10T20:20:16.100914+0000', 'last_active': '2024-09-10T20:31:12.731668+0000', 'last_peered': '2024-09-10T20:31:12.731668+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:31:12.731668+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1857}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:21.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-10T20:31:21.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "44'12291", 'reported_seq': 14988, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:19.223485+0000', 'last_change': '2024-09-10T20:31:11.673002+0000', 'last_active': '2024-09-10T20:31:19.223485+0000', 'last_peered': '2024-09-10T20:31:19.223485+0000', 'last_clean': '2024-09-10T20:19:06.164920+0000', 'last_became_active': '2024-09-10T20:20:16.138436+0000', 'last_became_peered': '2024-09-10T20:20:16.138436+0000', 'last_unstale': '2024-09-10T20:31:19.223485+0000', 'last_undegraded': '2024-09-10T20:20:15.494811+0000', 'last_fullsized': '2024-09-10T20:20:15.454837+0000', 'mapping_epoch': 31, 'log_start': "32'2200", 'ondisk_log_start': "32'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 747, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12289, 'num_write_kb': 49164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1300, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -745}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:31:21.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:21.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12401, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732241+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:12.732241+0000', 'last_peered': '2024-09-10T20:31:12.732241+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:12.732241+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:21.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:21.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12480, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.733198+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:12.733198+0000', 'last_peered': '2024-09-10T20:31:12.733198+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:12.733198+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:21.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:21.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12031, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732645+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:31:12.732645+0000', 'last_peered': '2024-09-10T20:31:12.732645+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:12.732645+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:21.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:22.295 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:22.296 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13849/44 objects degraded (31475.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":771,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11,"data_bytes":74096,"bytes_used":2745274368,"bytes_avail":383801782272,"bytes_total":386547056640,"degraded_objects":13849,"degraded_total":44,"degraded_ratio":314.75,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":244,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:22.308 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:31:22.309 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:22.612 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:22.612 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:22.625 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-10T20:31:22.144955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680932,"kb_used_data":1616,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806428,"statfs":{"total":386547056640,"available":383801782272,"internally_reserved":0,"allocated":1654784,"data_stored":649484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001722"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12422,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732630+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:12.732630+0000","last_peered":"2024-09-10T20:31:12.732630+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:12.732630+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735828+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:12.735828+0000","last_peered":"2024-09-10T20:31:12.735828+0000","last_clean":"2024-09-10T20:31:12.735828+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:12.735828+0000","last_undegraded":"2024-09-10T20:31:12.735828+0000","last_fullsized":"2024-09-10T20:31:12.735828+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12361,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732017+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:12.732017+0000","last_peered":"2024-09-10T20:31:12.732017+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:12.732017+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16594,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735089+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:12.735089+0000","last_peered":"2024-09-10T20:31:12.735089+0000","last_clean":"2024-09-10T20:31:12.735089+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:12.735089+0000","last_undegraded":"2024-09-10T20:31:12.735089+0000","last_fullsized":"2024-09-10T20:31:12.735089+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731714+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:12.731714+0000","last_peered":"2024-09-10T20:31:12.731714+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:12.731714+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12338,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731668+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:12.731668+0000","last_peered":"2024-09-10T20:31:12.731668+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:12.731668+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16097,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731237+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:12.731237+0000","last_peered":"2024-09-10T20:31:12.731237+0000","last_clean":"2024-09-10T20:31:12.731237+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:12.731237+0000","last_undegraded":"2024-09-10T20:31:12.731237+0000","last_fullsized":"2024-09-10T20:31:12.731237+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":14988,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:19.223485+0000","last_change":"2024-09-10T20:31:11.673002+0000","last_active":"2024-09-10T20:31:19.223485+0000","last_peered":"2024-09-10T20:31:19.223485+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:19.223485+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1300,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731530+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:12.731530+0000","last_peered":"2024-09-10T20:31:12.731530+0000","last_clean":"2024-09-10T20:31:12.731530+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:12.731530+0000","last_undegraded":"2024-09-10T20:31:12.731530+0000","last_fullsized":"2024-09-10T20:31:12.731530+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16130,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.732926+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:12.732926+0000","last_peered":"2024-09-10T20:31:12.732926+0000","last_clean":"2024-09-10T20:31:12.732926+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:12.732926+0000","last_undegraded":"2024-09-10T20:31:12.732926+0000","last_fullsized":"2024-09-10T20:31:12.732926+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12401,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732241+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:12.732241+0000","last_peered":"2024-09-10T20:31:12.732241+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:12.732241+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12480,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.733198+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:12.733198+0000","last_peered":"2024-09-10T20:31:12.733198+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:12.733198+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12031,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732645+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:12.732645+0000","last_peered":"2024-09-10T20:31:12.732645+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:12.732645+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.736102+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:12.736102+0000","last_peered":"2024-09-10T20:31:12.736102+0000","last_clean":"2024-09-10T20:31:12.736102+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:12.736102+0000","last_undegraded":"2024-09-10T20:31:12.736102+0000","last_fullsized":"2024-09-10T20:31:12.736102+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12685,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:13.134763+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:13.134763+0000","last_peered":"2024-09-10T20:31:13.134763+0000","last_clean":"2024-09-10T20:31:13.134763+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:13.134763+0000","last_undegraded":"2024-09-10T20:31:13.134763+0000","last_fullsized":"2024-09-10T20:31:13.134763+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16236,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.734331+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:12.734331+0000","last_peered":"2024-09-10T20:31:12.734331+0000","last_clean":"2024-09-10T20:31:12.734331+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:12.734331+0000","last_undegraded":"2024-09-10T20:31:12.734331+0000","last_fullsized":"2024-09-10T20:31:12.734331+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703520,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668320,"statfs":{"total":96636764160,"available":95916359680,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577724,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794116,"statfs":{"total":96636764160,"available":96045174784,"internally_reserved":0,"allocated":438272,"data_stored":186947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699812,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672028,"statfs":{"total":96636764160,"available":95920156672,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:25.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:25.931 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:25.931 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:25.943 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-10T20:31:24.145414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680932,"kb_used_data":1616,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806428,"statfs":{"total":386547056640,"available":383801782272,"internally_reserved":0,"allocated":1654784,"data_stored":649484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":899,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001556"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12422,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732630+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:12.732630+0000","last_peered":"2024-09-10T20:31:12.732630+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:12.732630+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735828+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:12.735828+0000","last_peered":"2024-09-10T20:31:12.735828+0000","last_clean":"2024-09-10T20:31:12.735828+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:12.735828+0000","last_undegraded":"2024-09-10T20:31:12.735828+0000","last_fullsized":"2024-09-10T20:31:12.735828+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12361,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732017+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:12.732017+0000","last_peered":"2024-09-10T20:31:12.732017+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:12.732017+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16594,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.735089+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:12.735089+0000","last_peered":"2024-09-10T20:31:12.735089+0000","last_clean":"2024-09-10T20:31:12.735089+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:12.735089+0000","last_undegraded":"2024-09-10T20:31:12.735089+0000","last_fullsized":"2024-09-10T20:31:12.735089+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731714+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:12.731714+0000","last_peered":"2024-09-10T20:31:12.731714+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:12.731714+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":12338,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.731668+0000","last_change":"2024-09-10T20:20:16.100914+0000","last_active":"2024-09-10T20:31:12.731668+0000","last_peered":"2024-09-10T20:31:12.731668+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:12.731668+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16097,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731237+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:12.731237+0000","last_peered":"2024-09-10T20:31:12.731237+0000","last_clean":"2024-09-10T20:31:12.731237+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:12.731237+0000","last_undegraded":"2024-09-10T20:31:12.731237+0000","last_fullsized":"2024-09-10T20:31:12.731237+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":14988,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:19.223485+0000","last_change":"2024-09-10T20:31:11.673002+0000","last_active":"2024-09-10T20:31:19.223485+0000","last_peered":"2024-09-10T20:31:19.223485+0000","last_clean":"2024-09-10T20:19:06.164920+0000","last_became_active":"2024-09-10T20:20:16.138436+0000","last_became_peered":"2024-09-10T20:20:16.138436+0000","last_unstale":"2024-09-10T20:31:19.223485+0000","last_undegraded":"2024-09-10T20:20:15.494811+0000","last_fullsized":"2024-09-10T20:20:15.454837+0000","mapping_epoch":31,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1300,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.731530+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:12.731530+0000","last_peered":"2024-09-10T20:31:12.731530+0000","last_clean":"2024-09-10T20:31:12.731530+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:12.731530+0000","last_undegraded":"2024-09-10T20:31:12.731530+0000","last_fullsized":"2024-09-10T20:31:12.731530+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16131,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.732978+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:12.732978+0000","last_peered":"2024-09-10T20:31:12.732978+0000","last_clean":"2024-09-10T20:31:12.732978+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:12.732978+0000","last_undegraded":"2024-09-10T20:31:12.732978+0000","last_fullsized":"2024-09-10T20:31:12.732978+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12401,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732241+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:12.732241+0000","last_peered":"2024-09-10T20:31:12.732241+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:12.732241+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12480,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.733198+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:12.733198+0000","last_peered":"2024-09-10T20:31:12.733198+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:12.733198+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12031,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732645+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:12.732645+0000","last_peered":"2024-09-10T20:31:12.732645+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:12.732645+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.736102+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:12.736102+0000","last_peered":"2024-09-10T20:31:12.736102+0000","last_clean":"2024-09-10T20:31:12.736102+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:12.736102+0000","last_undegraded":"2024-09-10T20:31:12.736102+0000","last_fullsized":"2024-09-10T20:31:12.736102+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12685,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:13.134763+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:13.134763+0000","last_peered":"2024-09-10T20:31:13.134763+0000","last_clean":"2024-09-10T20:31:13.134763+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:13.134763+0000","last_undegraded":"2024-09-10T20:31:13.134763+0000","last_fullsized":"2024-09-10T20:31:13.134763+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16236,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.734331+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:12.734331+0000","last_peered":"2024-09-10T20:31:12.734331+0000","last_clean":"2024-09-10T20:31:12.734331+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:12.734331+0000","last_undegraded":"2024-09-10T20:31:12.734331+0000","last_fullsized":"2024-09-10T20:31:12.734331+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74096,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129644,"num_write_kb":432434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703520,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668320,"statfs":{"total":96636764160,"available":95916359680,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577724,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794116,"statfs":{"total":96636764160,"available":96045174784,"internally_reserved":0,"allocated":438272,"data_stored":186947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699812,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672028,"statfs":{"total":96636764160,"available":95920156672,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:25.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:26.293 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:26.294 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:26.306 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-10T20:31:26.145815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672660,"kb_used_data":1664,"kb_used_omap":14,"kb_used_meta":2670897,"kb_avail":374814700,"statfs":{"total":386547056640,"available":383810252800,"internally_reserved":0,"allocated":1703936,"data_stored":669383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2734999094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12289,"num_write_kb":-49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001596"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12361,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732017+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:12.732017+0000","last_peered":"2024-09-10T20:31:12.732017+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:12.732017+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":13443,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:25.776489+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:25.776489+0000","last_peered":"2024-09-10T20:31:25.776489+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:25.776489+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16131,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.732978+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:12.732978+0000","last_peered":"2024-09-10T20:31:12.732978+0000","last_clean":"2024-09-10T20:31:12.732978+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:12.732978+0000","last_undegraded":"2024-09-10T20:31:12.732978+0000","last_fullsized":"2024-09-10T20:31:12.732978+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12031,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732645+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:12.732645+0000","last_peered":"2024-09-10T20:31:12.732645+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:12.732645+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703520,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668320,"statfs":{"total":96636764160,"available":95916359680,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569424,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93802416,"statfs":{"total":96636764160,"available":96053673984,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582610642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:26.307 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:26.307 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12426, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807669+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:23.807669+0000', 'last_peered': '2024-09-10T20:31:23.807669+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:23.807669+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:26.307 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12361, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732017+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:12.732017+0000', 'last_peered': '2024-09-10T20:31:12.732017+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:12.732017+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12509, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807493+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:23.807493+0000', 'last_peered': '2024-09-10T20:31:23.807493+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:23.807493+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 13443, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:25.776489+0000', 'last_change': '2024-09-10T20:31:22.492025+0000', 'last_active': '2024-09-10T20:31:25.776489+0000', 'last_peered': '2024-09-10T20:31:25.776489+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:31:25.776489+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 631, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1307}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12405, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807624+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:23.807624+0000', 'last_peered': '2024-09-10T20:31:23.807624+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:23.807624+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12484, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.809223+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:23.809223+0000', 'last_peered': '2024-09-10T20:31:23.809223+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:23.809223+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:26.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:26.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12031, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:12.732645+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:31:12.732645+0000', 'last_peered': '2024-09-10T20:31:12.732645+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:12.732645+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:26.309 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:26.683 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13849/44 objects degraded (31475.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":775,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11,"data_bytes":74096,"bytes_used":2745274368,"bytes_avail":383801782272,"bytes_total":386547056640,"degraded_objects":13849,"degraded_total":44,"degraded_ratio":314.75,"recovering_objects_per_sec":112,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":899,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:26.696 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:31:26.696 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:27.056 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:27.056 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:27.069 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-10T20:31:26.145815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672660,"kb_used_data":1664,"kb_used_omap":14,"kb_used_meta":2670897,"kb_avail":374814700,"statfs":{"total":386547056640,"available":383810252800,"internally_reserved":0,"allocated":1703936,"data_stored":669383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2734999094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12289,"num_write_kb":-49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001596"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12361,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732017+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:12.732017+0000","last_peered":"2024-09-10T20:31:12.732017+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:12.732017+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":13443,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:25.776489+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:25.776489+0000","last_peered":"2024-09-10T20:31:25.776489+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:25.776489+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16131,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-10T20:31:12.732978+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:12.732978+0000","last_peered":"2024-09-10T20:31:12.732978+0000","last_clean":"2024-09-10T20:31:12.732978+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:12.732978+0000","last_undegraded":"2024-09-10T20:31:12.732978+0000","last_fullsized":"2024-09-10T20:31:12.732978+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12031,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:12.732645+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:12.732645+0000","last_peered":"2024-09-10T20:31:12.732645+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:12.732645+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703520,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668320,"statfs":{"total":96636764160,"available":95916359680,"internally_reserved":0,"allocated":405504,"data_stored":154179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569424,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93802416,"statfs":{"total":96636764160,"available":96053673984,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582610642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:30.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:30.385 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:30.385 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:30.397 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-10T20:31:30.146523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680988,"kb_used_data":1680,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806372,"statfs":{"total":386547056640,"available":383801724928,"internally_reserved":0,"allocated":1720320,"data_stored":676016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12289,"num_write_kb":-49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-750,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001568"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804978+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:23.804978+0000","last_peered":"2024-09-10T20:31:23.804978+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:23.804978+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":13443,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:25.776489+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:25.776489+0000","last_peered":"2024-09-10T20:31:25.776489+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:25.776489+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:23.805043+0000","last_peered":"2024-09-10T20:31:23.805043+0000","last_clean":"2024-09-10T20:31:23.805043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:23.805043+0000","last_undegraded":"2024-09-10T20:31:23.805043+0000","last_fullsized":"2024-09-10T20:31:23.805043+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12035,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804410+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:23.804410+0000","last_peered":"2024-09-10T20:31:23.804410+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:23.804410+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703532,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668308,"statfs":{"total":96636764160,"available":95916347392,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577740,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794100,"statfs":{"total":96636764160,"available":96045158400,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:30.398 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:30.722 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:30.723 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:30.735 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-10T20:31:30.146523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680988,"kb_used_data":1680,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806372,"statfs":{"total":386547056640,"available":383801724928,"internally_reserved":0,"allocated":1720320,"data_stored":676016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12289,"num_write_kb":-49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-750,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001568"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804978+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:23.804978+0000","last_peered":"2024-09-10T20:31:23.804978+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:23.804978+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":13443,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:25.776489+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:25.776489+0000","last_peered":"2024-09-10T20:31:25.776489+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:25.776489+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:23.805043+0000","last_peered":"2024-09-10T20:31:23.805043+0000","last_clean":"2024-09-10T20:31:23.805043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:23.805043+0000","last_undegraded":"2024-09-10T20:31:23.805043+0000","last_fullsized":"2024-09-10T20:31:23.805043+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12035,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804410+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:23.804410+0000","last_peered":"2024-09-10T20:31:23.804410+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:23.804410+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703532,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668308,"statfs":{"total":96636764160,"available":95916347392,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577740,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794100,"statfs":{"total":96636764160,"available":96045158400,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12426, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807669+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:23.807669+0000', 'last_peered': '2024-09-10T20:31:23.807669+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:23.807669+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12365, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.804978+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:23.804978+0000', 'last_peered': '2024-09-10T20:31:23.804978+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:23.804978+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12509, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807493+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:23.807493+0000', 'last_peered': '2024-09-10T20:31:23.807493+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:23.807493+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:30.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 13443, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:25.776489+0000', 'last_change': '2024-09-10T20:31:22.492025+0000', 'last_active': '2024-09-10T20:31:25.776489+0000', 'last_peered': '2024-09-10T20:31:25.776489+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:31:25.776489+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 631, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1307}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:30.737 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:30.737 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12405, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807624+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:23.807624+0000', 'last_peered': '2024-09-10T20:31:23.807624+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:23.807624+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:30.737 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:30.737 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12484, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.809223+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:23.809223+0000', 'last_peered': '2024-09-10T20:31:23.809223+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:23.809223+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:30.737 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:30.737 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12035, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.804410+0000', 'last_change': '2024-09-10T20:20:16.118062+0000', 'last_active': '2024-09-10T20:31:23.804410+0000', 'last_peered': '2024-09-10T20:31:23.804410+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:23.804410+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:30.738 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:31.111 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:31.111 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12552/36 objects degraded (34866.667%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":779,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2745331712,"bytes_avail":383801724928,"bytes_total":386547056640,"degraded_objects":12552,"degraded_total":36,"degraded_ratio":348.66666666666674},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:31.125 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:31:31.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:31.406 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:31.407 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:31.418 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-10T20:31:30.146523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680988,"kb_used_data":1680,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806372,"statfs":{"total":386547056640,"available":383801724928,"internally_reserved":0,"allocated":1720320,"data_stored":676016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12289,"num_write_kb":-49164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-750,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001568"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804978+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:23.804978+0000","last_peered":"2024-09-10T20:31:23.804978+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:23.804978+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":13443,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:25.776489+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:25.776489+0000","last_peered":"2024-09-10T20:31:25.776489+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:25.776489+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:23.805043+0000","last_peered":"2024-09-10T20:31:23.805043+0000","last_clean":"2024-09-10T20:31:23.805043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:23.805043+0000","last_undegraded":"2024-09-10T20:31:23.805043+0000","last_fullsized":"2024-09-10T20:31:23.805043+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12035,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804410+0000","last_change":"2024-09-10T20:20:16.118062+0000","last_active":"2024-09-10T20:31:23.804410+0000","last_peered":"2024-09-10T20:31:23.804410+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:23.804410+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703532,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668308,"statfs":{"total":96636764160,"available":95916347392,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577740,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794100,"statfs":{"total":96636764160,"available":96045158400,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:34.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:34.717 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:34.717 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:34.729 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-10T20:31:34.147180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680988,"kb_used_data":1680,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806372,"statfs":{"total":386547056640,"available":383801724928,"internally_reserved":0,"allocated":1720320,"data_stored":676016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001365"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804978+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:23.804978+0000","last_peered":"2024-09-10T20:31:23.804978+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:23.804978+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":15389,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:30.779915+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:30.779915+0000","last_peered":"2024-09-10T20:31:30.779915+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:30.779915+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:23.805043+0000","last_peered":"2024-09-10T20:31:23.805043+0000","last_clean":"2024-09-10T20:31:23.805043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:23.805043+0000","last_undegraded":"2024-09-10T20:31:23.805043+0000","last_fullsized":"2024-09-10T20:31:23.805043+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12046,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:32.375919+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:32.375919+0000","last_peered":"2024-09-10T20:31:32.375919+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:32.375919+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703532,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668308,"statfs":{"total":96636764160,"available":95916347392,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577740,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794100,"statfs":{"total":96636764160,"available":96045158400,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:34.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:35.039 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:35.040 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:35.051 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-10T20:31:34.147180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680988,"kb_used_data":1680,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806372,"statfs":{"total":386547056640,"available":383801724928,"internally_reserved":0,"allocated":1720320,"data_stored":676016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001365"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804978+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:23.804978+0000","last_peered":"2024-09-10T20:31:23.804978+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:23.804978+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":15389,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:30.779915+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:30.779915+0000","last_peered":"2024-09-10T20:31:30.779915+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:30.779915+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:23.805043+0000","last_peered":"2024-09-10T20:31:23.805043+0000","last_clean":"2024-09-10T20:31:23.805043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:23.805043+0000","last_undegraded":"2024-09-10T20:31:23.805043+0000","last_fullsized":"2024-09-10T20:31:23.805043+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12046,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:32.375919+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:32.375919+0000","last_peered":"2024-09-10T20:31:32.375919+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:32.375919+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703532,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668308,"statfs":{"total":96636764160,"available":95916347392,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577740,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794100,"statfs":{"total":96636764160,"available":96045158400,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12426, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807669+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:23.807669+0000', 'last_peered': '2024-09-10T20:31:23.807669+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:23.807669+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12365, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.804978+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:23.804978+0000', 'last_peered': '2024-09-10T20:31:23.804978+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:23.804978+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12509, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807493+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:23.807493+0000', 'last_peered': '2024-09-10T20:31:23.807493+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:23.807493+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "44'12098", 'reported_seq': 15389, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:30.779915+0000', 'last_change': '2024-09-10T20:31:22.492025+0000', 'last_active': '2024-09-10T20:31:30.779915+0000', 'last_peered': '2024-09-10T20:31:30.779915+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.093143+0000', 'last_became_peered': '2024-09-10T20:20:16.093143+0000', 'last_unstale': '2024-09-10T20:31:30.779915+0000', 'last_undegraded': '2024-09-10T20:20:15.513034+0000', 'last_fullsized': '2024-09-10T20:20:15.455703+0000', 'mapping_epoch': 31, 'log_start': "32'2000", 'ondisk_log_start': "32'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 334, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 48392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1604, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -334}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:35.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:35.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12405, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.807624+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:23.807624+0000', 'last_peered': '2024-09-10T20:31:23.807624+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:23.807624+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:35.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:35.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12484, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:23.809223+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:23.809223+0000', 'last_peered': '2024-09-10T20:31:23.809223+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:23.809223+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:35.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:35.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 12046, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:32.375919+0000', 'last_change': '2024-09-10T20:31:32.340824+0000', 'last_active': '2024-09-10T20:31:32.375919+0000', 'last_peered': '2024-09-10T20:31:32.375919+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:32.375919+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:35.054 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:35.492 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:35.492 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11574/36 objects degraded (32150.000%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":784,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2745331712,"bytes_avail":383801724928,"bytes_total":386547056640,"degraded_objects":11574,"degraded_total":36,"degraded_ratio":321.5,"recovering_objects_per_sec":122,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":978,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:35.506 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:31:35.506 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:35.820 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:35.820 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:35.833 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-10T20:31:34.147180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680988,"kb_used_data":1680,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806372,"statfs":{"total":386547056640,"available":383801724928,"internally_reserved":0,"allocated":1720320,"data_stored":676016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001365"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12426,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807669+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:23.807669+0000","last_peered":"2024-09-10T20:31:23.807669+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:23.807669+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12406,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805647+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:23.805647+0000","last_peered":"2024-09-10T20:31:23.805647+0000","last_clean":"2024-09-10T20:31:23.805647+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:23.805647+0000","last_undegraded":"2024-09-10T20:31:23.805647+0000","last_fullsized":"2024-09-10T20:31:23.805647+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.804978+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:23.804978+0000","last_peered":"2024-09-10T20:31:23.804978+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:23.804978+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16598,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804801+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:23.804801+0000","last_peered":"2024-09-10T20:31:23.804801+0000","last_clean":"2024-09-10T20:31:23.804801+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:23.804801+0000","last_undegraded":"2024-09-10T20:31:23.804801+0000","last_fullsized":"2024-09-10T20:31:23.804801+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12509,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807493+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:23.807493+0000","last_peered":"2024-09-10T20:31:23.807493+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:23.807493+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":15389,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:30.779915+0000","last_change":"2024-09-10T20:31:22.492025+0000","last_active":"2024-09-10T20:31:30.779915+0000","last_peered":"2024-09-10T20:31:30.779915+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.093143+0000","last_became_peered":"2024-09-10T20:20:16.093143+0000","last_unstale":"2024-09-10T20:31:30.779915+0000","last_undegraded":"2024-09-10T20:20:15.513034+0000","last_fullsized":"2024-09-10T20:20:15.455703+0000","mapping_epoch":31,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804037+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:23.804037+0000","last_peered":"2024-09-10T20:31:23.804037+0000","last_clean":"2024-09-10T20:31:23.804037+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:23.804037+0000","last_undegraded":"2024-09-10T20:31:23.804037+0000","last_fullsized":"2024-09-10T20:31:23.804037+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12492,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:24.089308+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:24.089308+0000","last_peered":"2024-09-10T20:31:24.089308+0000","last_clean":"2024-09-10T20:31:24.089308+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:24.089308+0000","last_undegraded":"2024-09-10T20:31:24.089308+0000","last_fullsized":"2024-09-10T20:31:24.089308+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15994,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.807572+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:23.807572+0000","last_peered":"2024-09-10T20:31:23.807572+0000","last_clean":"2024-09-10T20:31:23.807572+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:23.807572+0000","last_undegraded":"2024-09-10T20:31:23.807572+0000","last_fullsized":"2024-09-10T20:31:23.807572+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.805043+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:23.805043+0000","last_peered":"2024-09-10T20:31:23.805043+0000","last_clean":"2024-09-10T20:31:23.805043+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:23.805043+0000","last_undegraded":"2024-09-10T20:31:23.805043+0000","last_fullsized":"2024-09-10T20:31:23.805043+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12405,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.807624+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:23.807624+0000","last_peered":"2024-09-10T20:31:23.807624+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:23.807624+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12484,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:23.809223+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:23.809223+0000","last_peered":"2024-09-10T20:31:23.809223+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:23.809223+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12046,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:32.375919+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:32.375919+0000","last_peered":"2024-09-10T20:31:32.375919+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:32.375919+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16164,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804652+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:23.804652+0000","last_peered":"2024-09-10T20:31:23.804652+0000","last_clean":"2024-09-10T20:31:23.804652+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:23.804652+0000","last_undegraded":"2024-09-10T20:31:23.804652+0000","last_fullsized":"2024-09-10T20:31:23.804652+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12689,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.804719+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:23.804719+0000","last_peered":"2024-09-10T20:31:23.804719+0000","last_clean":"2024-09-10T20:31:23.804719+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:23.804719+0000","last_undegraded":"2024-09-10T20:31:23.804719+0000","last_fullsized":"2024-09-10T20:31:23.804719+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16241,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-10T20:31:23.806176+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:23.806176+0000","last_peered":"2024-09-10T20:31:23.806176+0000","last_clean":"2024-09-10T20:31:23.806176+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:23.806176+0000","last_undegraded":"2024-09-10T20:31:23.806176+0000","last_fullsized":"2024-09-10T20:31:23.806176+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699888,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671952,"statfs":{"total":96636764160,"available":95920078848,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703532,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668308,"statfs":{"total":96636764160,"available":95916347392,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577740,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794100,"statfs":{"total":96636764160,"available":96045158400,"internally_reserved":0,"allocated":454656,"data_stored":193580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699828,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93672012,"statfs":{"total":96636764160,"available":95920140288,"internally_reserved":0,"allocated":421888,"data_stored":160812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:38.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:39.134 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:39.134 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:39.146 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-10T20:31:38.147851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672744,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2670897,"kb_avail":374814616,"statfs":{"total":386547056640,"available":383810166784,"internally_reserved":0,"allocated":1781760,"data_stored":698156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2734999094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001327"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12430,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850237+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:33.850237+0000","last_peered":"2024-09-10T20:31:33.850237+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:33.850237+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12410,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852001+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:33.852001+0000","last_peered":"2024-09-10T20:31:33.852001+0000","last_clean":"2024-09-10T20:31:33.852001+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:33.852001+0000","last_undegraded":"2024-09-10T20:31:33.852001+0000","last_fullsized":"2024-09-10T20:31:33.852001+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12369,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.852275+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:33.852275+0000","last_peered":"2024-09-10T20:31:33.852275+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:33.852275+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16602,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851767+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:33.851767+0000","last_peered":"2024-09-10T20:31:33.851767+0000","last_clean":"2024-09-10T20:31:33.851767+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:33.851767+0000","last_undegraded":"2024-09-10T20:31:33.851767+0000","last_fullsized":"2024-09-10T20:31:33.851767+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12513,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.849838+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:33.849838+0000","last_peered":"2024-09-10T20:31:33.849838+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:33.849838+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.913927+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:33.913927+0000","last_peered":"2024-09-10T20:31:33.913927+0000","last_clean":"2024-09-10T20:31:33.913927+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:33.913927+0000","last_undegraded":"2024-09-10T20:31:33.913927+0000","last_fullsized":"2024-09-10T20:31:33.913927+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.849301+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:33.849301+0000","last_peered":"2024-09-10T20:31:33.849301+0000","last_clean":"2024-09-10T20:31:33.849301+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:33.849301+0000","last_undegraded":"2024-09-10T20:31:33.849301+0000","last_fullsized":"2024-09-10T20:31:33.849301+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12496,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851556+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:33.851556+0000","last_peered":"2024-09-10T20:31:33.851556+0000","last_clean":"2024-09-10T20:31:33.851556+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:33.851556+0000","last_undegraded":"2024-09-10T20:31:33.851556+0000","last_fullsized":"2024-09-10T20:31:33.851556+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15998,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.850285+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:33.850285+0000","last_peered":"2024-09-10T20:31:33.850285+0000","last_clean":"2024-09-10T20:31:33.850285+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:33.850285+0000","last_undegraded":"2024-09-10T20:31:33.850285+0000","last_fullsized":"2024-09-10T20:31:33.850285+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852318+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:33.852318+0000","last_peered":"2024-09-10T20:31:33.852318+0000","last_clean":"2024-09-10T20:31:33.852318+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:33.852318+0000","last_undegraded":"2024-09-10T20:31:33.852318+0000","last_fullsized":"2024-09-10T20:31:33.852318+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12409,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850274+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:33.850274+0000","last_peered":"2024-09-10T20:31:33.850274+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:33.850274+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12488,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850238+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:33.850238+0000","last_peered":"2024-09-10T20:31:33.850238+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:33.850238+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":13696,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:37.376208+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:37.376208+0000","last_peered":"2024-09-10T20:31:37.376208+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:37.376208+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852460+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:33.852460+0000","last_peered":"2024-09-10T20:31:33.852460+0000","last_clean":"2024-09-10T20:31:33.852460+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:33.852460+0000","last_undegraded":"2024-09-10T20:31:33.852460+0000","last_fullsized":"2024-09-10T20:31:33.852460+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12693,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851687+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:33.851687+0000","last_peered":"2024-09-10T20:31:33.851687+0000","last_clean":"2024-09-10T20:31:33.851687+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:33.851687+0000","last_undegraded":"2024-09-10T20:31:33.851687+0000","last_fullsized":"2024-09-10T20:31:33.851687+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851661+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:33.851661+0000","last_peered":"2024-09-10T20:31:33.851661+0000","last_clean":"2024-09-10T20:31:33.851661+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:33.851661+0000","last_undegraded":"2024-09-10T20:31:33.851661+0000","last_fullsized":"2024-09-10T20:31:33.851661+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699908,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671932,"statfs":{"total":96636764160,"available":95920058368,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703552,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668288,"statfs":{"total":96636764160,"available":95916326912,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569440,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93802400,"statfs":{"total":96636764160,"available":96053657600,"internally_reserved":0,"allocated":466944,"data_stored":196043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582610642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215255,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699844,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671996,"statfs":{"total":96636764160,"available":95920123904,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:39.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:39.450 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:39.462 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-10T20:31:38.147851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672744,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2670897,"kb_avail":374814616,"statfs":{"total":386547056640,"available":383810166784,"internally_reserved":0,"allocated":1781760,"data_stored":698156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2734999094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001327"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12430,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850237+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:33.850237+0000","last_peered":"2024-09-10T20:31:33.850237+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:33.850237+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12410,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852001+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:33.852001+0000","last_peered":"2024-09-10T20:31:33.852001+0000","last_clean":"2024-09-10T20:31:33.852001+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:33.852001+0000","last_undegraded":"2024-09-10T20:31:33.852001+0000","last_fullsized":"2024-09-10T20:31:33.852001+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12369,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.852275+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:33.852275+0000","last_peered":"2024-09-10T20:31:33.852275+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:33.852275+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16602,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851767+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:33.851767+0000","last_peered":"2024-09-10T20:31:33.851767+0000","last_clean":"2024-09-10T20:31:33.851767+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:33.851767+0000","last_undegraded":"2024-09-10T20:31:33.851767+0000","last_fullsized":"2024-09-10T20:31:33.851767+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12513,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.849838+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:33.849838+0000","last_peered":"2024-09-10T20:31:33.849838+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:33.849838+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.913927+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:33.913927+0000","last_peered":"2024-09-10T20:31:33.913927+0000","last_clean":"2024-09-10T20:31:33.913927+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:33.913927+0000","last_undegraded":"2024-09-10T20:31:33.913927+0000","last_fullsized":"2024-09-10T20:31:33.913927+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.849301+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:33.849301+0000","last_peered":"2024-09-10T20:31:33.849301+0000","last_clean":"2024-09-10T20:31:33.849301+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:33.849301+0000","last_undegraded":"2024-09-10T20:31:33.849301+0000","last_fullsized":"2024-09-10T20:31:33.849301+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12496,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851556+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:33.851556+0000","last_peered":"2024-09-10T20:31:33.851556+0000","last_clean":"2024-09-10T20:31:33.851556+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:33.851556+0000","last_undegraded":"2024-09-10T20:31:33.851556+0000","last_fullsized":"2024-09-10T20:31:33.851556+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15998,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.850285+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:33.850285+0000","last_peered":"2024-09-10T20:31:33.850285+0000","last_clean":"2024-09-10T20:31:33.850285+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:33.850285+0000","last_undegraded":"2024-09-10T20:31:33.850285+0000","last_fullsized":"2024-09-10T20:31:33.850285+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852318+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:33.852318+0000","last_peered":"2024-09-10T20:31:33.852318+0000","last_clean":"2024-09-10T20:31:33.852318+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:33.852318+0000","last_undegraded":"2024-09-10T20:31:33.852318+0000","last_fullsized":"2024-09-10T20:31:33.852318+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12409,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850274+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:33.850274+0000","last_peered":"2024-09-10T20:31:33.850274+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:33.850274+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12488,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850238+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:33.850238+0000","last_peered":"2024-09-10T20:31:33.850238+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:33.850238+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":13696,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:37.376208+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:37.376208+0000","last_peered":"2024-09-10T20:31:37.376208+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:37.376208+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852460+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:33.852460+0000","last_peered":"2024-09-10T20:31:33.852460+0000","last_clean":"2024-09-10T20:31:33.852460+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:33.852460+0000","last_undegraded":"2024-09-10T20:31:33.852460+0000","last_fullsized":"2024-09-10T20:31:33.852460+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12693,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851687+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:33.851687+0000","last_peered":"2024-09-10T20:31:33.851687+0000","last_clean":"2024-09-10T20:31:33.851687+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:33.851687+0000","last_undegraded":"2024-09-10T20:31:33.851687+0000","last_fullsized":"2024-09-10T20:31:33.851687+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851661+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:33.851661+0000","last_peered":"2024-09-10T20:31:33.851661+0000","last_clean":"2024-09-10T20:31:33.851661+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:33.851661+0000","last_undegraded":"2024-09-10T20:31:33.851661+0000","last_fullsized":"2024-09-10T20:31:33.851661+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699908,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671932,"statfs":{"total":96636764160,"available":95920058368,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703552,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668288,"statfs":{"total":96636764160,"available":95916326912,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569440,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93802400,"statfs":{"total":96636764160,"available":96053657600,"internally_reserved":0,"allocated":466944,"data_stored":196043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582610642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215255,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699844,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671996,"statfs":{"total":96636764160,"available":95920123904,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:39.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:39.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12430, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.850237+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:33.850237+0000', 'last_peered': '2024-09-10T20:31:33.850237+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:33.850237+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:39.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12369, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.852275+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:33.852275+0000', 'last_peered': '2024-09-10T20:31:33.852275+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:33.852275+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12513, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.849838+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:33.849838+0000', 'last_peered': '2024-09-10T20:31:33.849838+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:33.849838+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12409, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.850274+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:33.850274+0000', 'last_peered': '2024-09-10T20:31:33.850274+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:33.850274+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12488, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.850238+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:33.850238+0000', 'last_peered': '2024-09-10T20:31:33.850238+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:33.850238+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:39.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 13696, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:37.376208+0000', 'last_change': '2024-09-10T20:31:32.340824+0000', 'last_active': '2024-09-10T20:31:37.376208+0000', 'last_peered': '2024-09-10T20:31:37.376208+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:37.376208+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 874, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:39.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:39.872 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10417/36 objects degraded (28936.111%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":788,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2736889856,"bytes_avail":383810166784,"bytes_total":386547056640,"degraded_objects":10417,"degraded_total":36,"degraded_ratio":289.36111111111109,"recovering_objects_per_sec":266,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2135,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:39.885 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:31:39.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:40.202 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:40.215 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-10T20:31:40.148257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681056,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806304,"statfs":{"total":386547056640,"available":383801655296,"internally_reserved":0,"allocated":1781760,"data_stored":698156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001413"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12430,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850237+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:33.850237+0000","last_peered":"2024-09-10T20:31:33.850237+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:33.850237+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12410,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852001+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:33.852001+0000","last_peered":"2024-09-10T20:31:33.852001+0000","last_clean":"2024-09-10T20:31:33.852001+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:33.852001+0000","last_undegraded":"2024-09-10T20:31:33.852001+0000","last_fullsized":"2024-09-10T20:31:33.852001+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12369,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.852275+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:33.852275+0000","last_peered":"2024-09-10T20:31:33.852275+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:33.852275+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16602,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851767+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:33.851767+0000","last_peered":"2024-09-10T20:31:33.851767+0000","last_clean":"2024-09-10T20:31:33.851767+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:33.851767+0000","last_undegraded":"2024-09-10T20:31:33.851767+0000","last_fullsized":"2024-09-10T20:31:33.851767+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12513,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.849838+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:33.849838+0000","last_peered":"2024-09-10T20:31:33.849838+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:33.849838+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.913927+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:33.913927+0000","last_peered":"2024-09-10T20:31:33.913927+0000","last_clean":"2024-09-10T20:31:33.913927+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:33.913927+0000","last_undegraded":"2024-09-10T20:31:33.913927+0000","last_fullsized":"2024-09-10T20:31:33.913927+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.849301+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:33.849301+0000","last_peered":"2024-09-10T20:31:33.849301+0000","last_clean":"2024-09-10T20:31:33.849301+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:33.849301+0000","last_undegraded":"2024-09-10T20:31:33.849301+0000","last_fullsized":"2024-09-10T20:31:33.849301+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12496,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851556+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:33.851556+0000","last_peered":"2024-09-10T20:31:33.851556+0000","last_clean":"2024-09-10T20:31:33.851556+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:33.851556+0000","last_undegraded":"2024-09-10T20:31:33.851556+0000","last_fullsized":"2024-09-10T20:31:33.851556+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15998,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.850285+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:33.850285+0000","last_peered":"2024-09-10T20:31:33.850285+0000","last_clean":"2024-09-10T20:31:33.850285+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:33.850285+0000","last_undegraded":"2024-09-10T20:31:33.850285+0000","last_fullsized":"2024-09-10T20:31:33.850285+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852318+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:33.852318+0000","last_peered":"2024-09-10T20:31:33.852318+0000","last_clean":"2024-09-10T20:31:33.852318+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:33.852318+0000","last_undegraded":"2024-09-10T20:31:33.852318+0000","last_fullsized":"2024-09-10T20:31:33.852318+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12409,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850274+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:33.850274+0000","last_peered":"2024-09-10T20:31:33.850274+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:33.850274+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12488,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850238+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:33.850238+0000","last_peered":"2024-09-10T20:31:33.850238+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:33.850238+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":13696,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:37.376208+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:37.376208+0000","last_peered":"2024-09-10T20:31:37.376208+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:37.376208+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852460+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:33.852460+0000","last_peered":"2024-09-10T20:31:33.852460+0000","last_clean":"2024-09-10T20:31:33.852460+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:33.852460+0000","last_undegraded":"2024-09-10T20:31:33.852460+0000","last_fullsized":"2024-09-10T20:31:33.852460+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12693,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851687+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:33.851687+0000","last_peered":"2024-09-10T20:31:33.851687+0000","last_clean":"2024-09-10T20:31:33.851687+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:33.851687+0000","last_undegraded":"2024-09-10T20:31:33.851687+0000","last_fullsized":"2024-09-10T20:31:33.851687+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851661+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:33.851661+0000","last_peered":"2024-09-10T20:31:33.851661+0000","last_clean":"2024-09-10T20:31:33.851661+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:33.851661+0000","last_undegraded":"2024-09-10T20:31:33.851661+0000","last_fullsized":"2024-09-10T20:31:33.851661+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699908,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671932,"statfs":{"total":96636764160,"available":95920058368,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703552,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668288,"statfs":{"total":96636764160,"available":95916326912,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577752,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794088,"statfs":{"total":96636764160,"available":96045146112,"internally_reserved":0,"allocated":466944,"data_stored":196043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699844,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671996,"statfs":{"total":96636764160,"available":95920123904,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:43.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:43.525 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:43.536 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-10T20:31:42.875873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681056,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806304,"statfs":{"total":386547056640,"available":383801655296,"internally_reserved":0,"allocated":1781760,"data_stored":698156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.027796"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12430,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850237+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:33.850237+0000","last_peered":"2024-09-10T20:31:33.850237+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:33.850237+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12410,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852001+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:33.852001+0000","last_peered":"2024-09-10T20:31:33.852001+0000","last_clean":"2024-09-10T20:31:33.852001+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:33.852001+0000","last_undegraded":"2024-09-10T20:31:33.852001+0000","last_fullsized":"2024-09-10T20:31:33.852001+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12369,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.852275+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:33.852275+0000","last_peered":"2024-09-10T20:31:33.852275+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:33.852275+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16602,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851767+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:33.851767+0000","last_peered":"2024-09-10T20:31:33.851767+0000","last_clean":"2024-09-10T20:31:33.851767+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:33.851767+0000","last_undegraded":"2024-09-10T20:31:33.851767+0000","last_fullsized":"2024-09-10T20:31:33.851767+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12513,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.849838+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:33.849838+0000","last_peered":"2024-09-10T20:31:33.849838+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:33.849838+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.913927+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:33.913927+0000","last_peered":"2024-09-10T20:31:33.913927+0000","last_clean":"2024-09-10T20:31:33.913927+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:33.913927+0000","last_undegraded":"2024-09-10T20:31:33.913927+0000","last_fullsized":"2024-09-10T20:31:33.913927+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.849301+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:33.849301+0000","last_peered":"2024-09-10T20:31:33.849301+0000","last_clean":"2024-09-10T20:31:33.849301+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:33.849301+0000","last_undegraded":"2024-09-10T20:31:33.849301+0000","last_fullsized":"2024-09-10T20:31:33.849301+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12496,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851556+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:33.851556+0000","last_peered":"2024-09-10T20:31:33.851556+0000","last_clean":"2024-09-10T20:31:33.851556+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:33.851556+0000","last_undegraded":"2024-09-10T20:31:33.851556+0000","last_fullsized":"2024-09-10T20:31:33.851556+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15998,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.850285+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:33.850285+0000","last_peered":"2024-09-10T20:31:33.850285+0000","last_clean":"2024-09-10T20:31:33.850285+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:33.850285+0000","last_undegraded":"2024-09-10T20:31:33.850285+0000","last_fullsized":"2024-09-10T20:31:33.850285+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852318+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:33.852318+0000","last_peered":"2024-09-10T20:31:33.852318+0000","last_clean":"2024-09-10T20:31:33.852318+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:33.852318+0000","last_undegraded":"2024-09-10T20:31:33.852318+0000","last_fullsized":"2024-09-10T20:31:33.852318+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12409,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850274+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:33.850274+0000","last_peered":"2024-09-10T20:31:33.850274+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:33.850274+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12488,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850238+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:33.850238+0000","last_peered":"2024-09-10T20:31:33.850238+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:33.850238+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":15772,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:42.377917+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:42.377917+0000","last_peered":"2024-09-10T20:31:42.377917+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:42.377917+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-37}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852460+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:33.852460+0000","last_peered":"2024-09-10T20:31:33.852460+0000","last_clean":"2024-09-10T20:31:33.852460+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:33.852460+0000","last_undegraded":"2024-09-10T20:31:33.852460+0000","last_fullsized":"2024-09-10T20:31:33.852460+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12693,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851687+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:33.851687+0000","last_peered":"2024-09-10T20:31:33.851687+0000","last_clean":"2024-09-10T20:31:33.851687+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:33.851687+0000","last_undegraded":"2024-09-10T20:31:33.851687+0000","last_fullsized":"2024-09-10T20:31:33.851687+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851661+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:33.851661+0000","last_peered":"2024-09-10T20:31:33.851661+0000","last_clean":"2024-09-10T20:31:33.851661+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:33.851661+0000","last_undegraded":"2024-09-10T20:31:33.851661+0000","last_fullsized":"2024-09-10T20:31:33.851661+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699908,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671932,"statfs":{"total":96636764160,"available":95920058368,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703552,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668288,"statfs":{"total":96636764160,"available":95916326912,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577752,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794088,"statfs":{"total":96636764160,"available":96045146112,"internally_reserved":0,"allocated":466944,"data_stored":196043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699844,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671996,"statfs":{"total":96636764160,"available":95920123904,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:43.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:43.851 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:43.863 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-10T20:31:42.875873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681056,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806304,"statfs":{"total":386547056640,"available":383801655296,"internally_reserved":0,"allocated":1781760,"data_stored":698156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.027796"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12430,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850237+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:33.850237+0000","last_peered":"2024-09-10T20:31:33.850237+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:33.850237+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12410,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852001+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:33.852001+0000","last_peered":"2024-09-10T20:31:33.852001+0000","last_clean":"2024-09-10T20:31:33.852001+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:33.852001+0000","last_undegraded":"2024-09-10T20:31:33.852001+0000","last_fullsized":"2024-09-10T20:31:33.852001+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12369,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.852275+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:33.852275+0000","last_peered":"2024-09-10T20:31:33.852275+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:33.852275+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16602,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851767+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:33.851767+0000","last_peered":"2024-09-10T20:31:33.851767+0000","last_clean":"2024-09-10T20:31:33.851767+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:33.851767+0000","last_undegraded":"2024-09-10T20:31:33.851767+0000","last_fullsized":"2024-09-10T20:31:33.851767+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12513,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.849838+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:33.849838+0000","last_peered":"2024-09-10T20:31:33.849838+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:33.849838+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.913927+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:33.913927+0000","last_peered":"2024-09-10T20:31:33.913927+0000","last_clean":"2024-09-10T20:31:33.913927+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:33.913927+0000","last_undegraded":"2024-09-10T20:31:33.913927+0000","last_fullsized":"2024-09-10T20:31:33.913927+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.849301+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:33.849301+0000","last_peered":"2024-09-10T20:31:33.849301+0000","last_clean":"2024-09-10T20:31:33.849301+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:33.849301+0000","last_undegraded":"2024-09-10T20:31:33.849301+0000","last_fullsized":"2024-09-10T20:31:33.849301+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12496,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851556+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:33.851556+0000","last_peered":"2024-09-10T20:31:33.851556+0000","last_clean":"2024-09-10T20:31:33.851556+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:33.851556+0000","last_undegraded":"2024-09-10T20:31:33.851556+0000","last_fullsized":"2024-09-10T20:31:33.851556+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15998,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.850285+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:33.850285+0000","last_peered":"2024-09-10T20:31:33.850285+0000","last_clean":"2024-09-10T20:31:33.850285+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:33.850285+0000","last_undegraded":"2024-09-10T20:31:33.850285+0000","last_fullsized":"2024-09-10T20:31:33.850285+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852318+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:33.852318+0000","last_peered":"2024-09-10T20:31:33.852318+0000","last_clean":"2024-09-10T20:31:33.852318+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:33.852318+0000","last_undegraded":"2024-09-10T20:31:33.852318+0000","last_fullsized":"2024-09-10T20:31:33.852318+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12409,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850274+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:33.850274+0000","last_peered":"2024-09-10T20:31:33.850274+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:33.850274+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12488,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850238+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:33.850238+0000","last_peered":"2024-09-10T20:31:33.850238+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:33.850238+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":15772,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:42.377917+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:42.377917+0000","last_peered":"2024-09-10T20:31:42.377917+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:42.377917+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-37}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852460+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:33.852460+0000","last_peered":"2024-09-10T20:31:33.852460+0000","last_clean":"2024-09-10T20:31:33.852460+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:33.852460+0000","last_undegraded":"2024-09-10T20:31:33.852460+0000","last_fullsized":"2024-09-10T20:31:33.852460+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12693,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851687+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:33.851687+0000","last_peered":"2024-09-10T20:31:33.851687+0000","last_clean":"2024-09-10T20:31:33.851687+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:33.851687+0000","last_undegraded":"2024-09-10T20:31:33.851687+0000","last_fullsized":"2024-09-10T20:31:33.851687+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851661+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:33.851661+0000","last_peered":"2024-09-10T20:31:33.851661+0000","last_clean":"2024-09-10T20:31:33.851661+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:33.851661+0000","last_undegraded":"2024-09-10T20:31:33.851661+0000","last_fullsized":"2024-09-10T20:31:33.851661+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699908,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671932,"statfs":{"total":96636764160,"available":95920058368,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703552,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668288,"statfs":{"total":96636764160,"available":95916326912,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577752,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794088,"statfs":{"total":96636764160,"available":96045146112,"internally_reserved":0,"allocated":466944,"data_stored":196043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699844,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671996,"statfs":{"total":96636764160,"available":95920123904,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:43.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12430, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.850237+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:33.850237+0000', 'last_peered': '2024-09-10T20:31:33.850237+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:33.850237+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12369, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.852275+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:33.852275+0000', 'last_peered': '2024-09-10T20:31:33.852275+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:33.852275+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12513, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.849838+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:33.849838+0000', 'last_peered': '2024-09-10T20:31:33.849838+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:33.849838+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12409, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.850274+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:33.850274+0000', 'last_peered': '2024-09-10T20:31:33.850274+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:33.850274+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:43.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 12488, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:33.850238+0000', 'last_change': '2024-09-10T20:20:16.143143+0000', 'last_active': '2024-09-10T20:31:33.850238+0000', 'last_peered': '2024-09-10T20:31:33.850238+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:33.850238+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:43.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-10T20:31:43.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "44'11830", 'reported_seq': 15772, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:42.377917+0000', 'last_change': '2024-09-10T20:31:32.340824+0000', 'last_active': '2024-09-10T20:31:42.377917+0000', 'last_peered': '2024-09-10T20:31:42.377917+0000', 'last_clean': '2024-09-10T20:18:59.040652+0000', 'last_became_active': '2024-09-10T20:20:16.111683+0000', 'last_became_peered': '2024-09-10T20:20:16.111683+0000', 'last_unstale': '2024-09-10T20:31:42.377917+0000', 'last_undegraded': '2024-09-10T20:20:15.478951+0000', 'last_fullsized': '2024-09-10T20:20:15.457549+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 38, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11829, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1912, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -37}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:43.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:44.301 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:44.301 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10417/36 objects degraded (28936.111%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":793,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2745401344,"bytes_avail":383801655296,"bytes_total":386547056640,"degraded_objects":10417,"degraded_total":36,"degraded_ratio":289.36111111111109,"recovering_objects_per_sec":124,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1157,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-09-10T20:30:08.119839+0000","services":{}},"progress_events":{}} 2024-09-10T20:31:44.317 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:31:44.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:44.630 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:44.643 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-10T20:31:44.148878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2681056,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2679217,"kb_avail":374806304,"statfs":{"total":386547056640,"available":383801655296,"internally_reserved":0,"allocated":1781760,"data_stored":698156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743518774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001378"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12430,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850237+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:33.850237+0000","last_peered":"2024-09-10T20:31:33.850237+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:33.850237+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12410,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852001+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:33.852001+0000","last_peered":"2024-09-10T20:31:33.852001+0000","last_clean":"2024-09-10T20:31:33.852001+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:33.852001+0000","last_undegraded":"2024-09-10T20:31:33.852001+0000","last_fullsized":"2024-09-10T20:31:33.852001+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12369,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.852275+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:33.852275+0000","last_peered":"2024-09-10T20:31:33.852275+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:33.852275+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16602,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851767+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:33.851767+0000","last_peered":"2024-09-10T20:31:33.851767+0000","last_clean":"2024-09-10T20:31:33.851767+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:33.851767+0000","last_undegraded":"2024-09-10T20:31:33.851767+0000","last_fullsized":"2024-09-10T20:31:33.851767+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12513,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.849838+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:33.849838+0000","last_peered":"2024-09-10T20:31:33.849838+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:33.849838+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.913927+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:33.913927+0000","last_peered":"2024-09-10T20:31:33.913927+0000","last_clean":"2024-09-10T20:31:33.913927+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:33.913927+0000","last_undegraded":"2024-09-10T20:31:33.913927+0000","last_fullsized":"2024-09-10T20:31:33.913927+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.849301+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:33.849301+0000","last_peered":"2024-09-10T20:31:33.849301+0000","last_clean":"2024-09-10T20:31:33.849301+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:33.849301+0000","last_undegraded":"2024-09-10T20:31:33.849301+0000","last_fullsized":"2024-09-10T20:31:33.849301+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12496,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851556+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:33.851556+0000","last_peered":"2024-09-10T20:31:33.851556+0000","last_clean":"2024-09-10T20:31:33.851556+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:33.851556+0000","last_undegraded":"2024-09-10T20:31:33.851556+0000","last_fullsized":"2024-09-10T20:31:33.851556+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":15998,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.850285+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:33.850285+0000","last_peered":"2024-09-10T20:31:33.850285+0000","last_clean":"2024-09-10T20:31:33.850285+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:33.850285+0000","last_undegraded":"2024-09-10T20:31:33.850285+0000","last_fullsized":"2024-09-10T20:31:33.850285+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852318+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:33.852318+0000","last_peered":"2024-09-10T20:31:33.852318+0000","last_clean":"2024-09-10T20:31:33.852318+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:33.852318+0000","last_undegraded":"2024-09-10T20:31:33.852318+0000","last_fullsized":"2024-09-10T20:31:33.852318+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12409,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850274+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:33.850274+0000","last_peered":"2024-09-10T20:31:33.850274+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:33.850274+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12488,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.850238+0000","last_change":"2024-09-10T20:20:16.143143+0000","last_active":"2024-09-10T20:31:33.850238+0000","last_peered":"2024-09-10T20:31:33.850238+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:33.850238+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":15772,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:42.377917+0000","last_change":"2024-09-10T20:31:32.340824+0000","last_active":"2024-09-10T20:31:42.377917+0000","last_peered":"2024-09-10T20:31:42.377917+0000","last_clean":"2024-09-10T20:18:59.040652+0000","last_became_active":"2024-09-10T20:20:16.111683+0000","last_became_peered":"2024-09-10T20:20:16.111683+0000","last_unstale":"2024-09-10T20:31:42.377917+0000","last_undegraded":"2024-09-10T20:20:15.478951+0000","last_fullsized":"2024-09-10T20:20:15.457549+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-37}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852460+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:33.852460+0000","last_peered":"2024-09-10T20:31:33.852460+0000","last_clean":"2024-09-10T20:31:33.852460+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:33.852460+0000","last_undegraded":"2024-09-10T20:31:33.852460+0000","last_fullsized":"2024-09-10T20:31:33.852460+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12693,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851687+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:33.851687+0000","last_peered":"2024-09-10T20:31:33.851687+0000","last_clean":"2024-09-10T20:31:33.851687+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:33.851687+0000","last_undegraded":"2024-09-10T20:31:33.851687+0000","last_fullsized":"2024-09-10T20:31:33.851687+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.851661+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:33.851661+0000","last_peered":"2024-09-10T20:31:33.851661+0000","last_clean":"2024-09-10T20:31:33.851661+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:33.851661+0000","last_undegraded":"2024-09-10T20:31:33.851661+0000","last_fullsized":"2024-09-10T20:31:33.851661+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699908,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671932,"statfs":{"total":96636764160,"available":95920058368,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703552,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668288,"statfs":{"total":96636764160,"available":95916326912,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577752,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93794088,"statfs":{"total":96636764160,"available":96045146112,"internally_reserved":0,"allocated":466944,"data_stored":196043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591130322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699844,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671996,"statfs":{"total":96636764160,"available":95920123904,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:47.645 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:47.946 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:47.946 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-10T20:31:46.149237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690832,"kb_used_data":1784,"kb_used_omap":14,"kb_used_meta":2688945,"kb_avail":374796528,"statfs":{"total":386547056640,"available":383791644672,"internally_reserved":0,"allocated":1826816,"data_stored":713515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2753480246},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001385"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12434,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884857+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:43.884857+0000","last_peered":"2024-09-10T20:31:43.884857+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:43.884857+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12414,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:43.883311+0000","last_peered":"2024-09-10T20:31:43.883311+0000","last_clean":"2024-09-10T20:31:43.883311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:43.883311+0000","last_undegraded":"2024-09-10T20:31:43.883311+0000","last_fullsized":"2024-09-10T20:31:43.883311+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12369,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:33.852275+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:33.852275+0000","last_peered":"2024-09-10T20:31:33.852275+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:33.852275+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883006+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:43.883006+0000","last_peered":"2024-09-10T20:31:43.883006+0000","last_clean":"2024-09-10T20:31:43.883006+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:43.883006+0000","last_undegraded":"2024-09-10T20:31:43.883006+0000","last_fullsized":"2024-09-10T20:31:43.883006+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12517,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884761+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:43.884761+0000","last_peered":"2024-09-10T20:31:43.884761+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:43.884761+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16074,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885128+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:43.885128+0000","last_peered":"2024-09-10T20:31:43.885128+0000","last_clean":"2024-09-10T20:31:43.885128+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:43.885128+0000","last_undegraded":"2024-09-10T20:31:43.885128+0000","last_fullsized":"2024-09-10T20:31:43.885128+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.881097+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:43.881097+0000","last_peered":"2024-09-10T20:31:43.881097+0000","last_clean":"2024-09-10T20:31:43.881097+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:43.881097+0000","last_undegraded":"2024-09-10T20:31:43.881097+0000","last_fullsized":"2024-09-10T20:31:43.881097+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12500,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882772+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:43.882772+0000","last_peered":"2024-09-10T20:31:43.882772+0000","last_clean":"2024-09-10T20:31:43.882772+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:43.882772+0000","last_undegraded":"2024-09-10T20:31:43.882772+0000","last_fullsized":"2024-09-10T20:31:43.882772+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16002,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885008+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:43.885008+0000","last_peered":"2024-09-10T20:31:43.885008+0000","last_clean":"2024-09-10T20:31:43.885008+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:43.885008+0000","last_undegraded":"2024-09-10T20:31:43.885008+0000","last_fullsized":"2024-09-10T20:31:43.885008+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-10T20:31:33.852318+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:33.852318+0000","last_peered":"2024-09-10T20:31:33.852318+0000","last_clean":"2024-09-10T20:31:33.852318+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:33.852318+0000","last_undegraded":"2024-09-10T20:31:33.852318+0000","last_fullsized":"2024-09-10T20:31:33.852318+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12413,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885200+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:43.885200+0000","last_peered":"2024-09-10T20:31:43.885200+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:43.885200+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":13521,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:45.777437+0000","last_change":"2024-09-10T20:31:42.530238+0000","last_active":"2024-09-10T20:31:45.777437+0000","last_peered":"2024-09-10T20:31:45.777437+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:45.777437+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.961301+0000","last_change":"2024-09-10T20:31:43.960747+0000","last_active":"2024-09-10T20:31:43.961301+0000","last_peered":"2024-09-10T20:31:43.961301+0000","last_clean":"2024-09-10T20:31:43.961301+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:43.961301+0000","last_undegraded":"2024-09-10T20:31:43.961301+0000","last_fullsized":"2024-09-10T20:31:43.961301+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16173,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883190+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:43.883190+0000","last_peered":"2024-09-10T20:31:43.883190+0000","last_clean":"2024-09-10T20:31:43.883190+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:43.883190+0000","last_undegraded":"2024-09-10T20:31:43.883190+0000","last_fullsized":"2024-09-10T20:31:43.883190+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882965+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:43.882965+0000","last_peered":"2024-09-10T20:31:43.882965+0000","last_clean":"2024-09-10T20:31:43.882965+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:43.882965+0000","last_undegraded":"2024-09-10T20:31:43.882965+0000","last_fullsized":"2024-09-10T20:31:43.882965+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16249,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882885+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:43.882885+0000","last_peered":"2024-09-10T20:31:43.882885+0000","last_clean":"2024-09-10T20:31:43.882885+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:43.882885+0000","last_undegraded":"2024-09-10T20:31:43.882885+0000","last_fullsized":"2024-09-10T20:31:43.882885+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699924,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671916,"statfs":{"total":96636764160,"available":95920041984,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703552,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668288,"statfs":{"total":96636764160,"available":95916326912,"internally_reserved":0,"allocated":438272,"data_stored":167371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587500,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784340,"statfs":{"total":96636764160,"available":96035164160,"internally_reserved":0,"allocated":479232,"data_stored":198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:47.959 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:48.268 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:48.269 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:48.281 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-10T20:31:48.149530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690852,"kb_used_data":1800,"kb_used_omap":14,"kb_used_meta":2688945,"kb_avail":374796508,"statfs":{"total":386547056640,"available":383791624192,"internally_reserved":0,"allocated":1843200,"data_stored":720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2753480246},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001273"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12434,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884857+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:43.884857+0000","last_peered":"2024-09-10T20:31:43.884857+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:43.884857+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12414,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:43.883311+0000","last_peered":"2024-09-10T20:31:43.883311+0000","last_clean":"2024-09-10T20:31:43.883311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:43.883311+0000","last_undegraded":"2024-09-10T20:31:43.883311+0000","last_fullsized":"2024-09-10T20:31:43.883311+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883006+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:43.883006+0000","last_peered":"2024-09-10T20:31:43.883006+0000","last_clean":"2024-09-10T20:31:43.883006+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:43.883006+0000","last_undegraded":"2024-09-10T20:31:43.883006+0000","last_fullsized":"2024-09-10T20:31:43.883006+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12517,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884761+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:43.884761+0000","last_peered":"2024-09-10T20:31:43.884761+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:43.884761+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16074,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885128+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:43.885128+0000","last_peered":"2024-09-10T20:31:43.885128+0000","last_clean":"2024-09-10T20:31:43.885128+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:43.885128+0000","last_undegraded":"2024-09-10T20:31:43.885128+0000","last_fullsized":"2024-09-10T20:31:43.885128+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.881097+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:43.881097+0000","last_peered":"2024-09-10T20:31:43.881097+0000","last_clean":"2024-09-10T20:31:43.881097+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:43.881097+0000","last_undegraded":"2024-09-10T20:31:43.881097+0000","last_fullsized":"2024-09-10T20:31:43.881097+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12500,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882772+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:43.882772+0000","last_peered":"2024-09-10T20:31:43.882772+0000","last_clean":"2024-09-10T20:31:43.882772+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:43.882772+0000","last_undegraded":"2024-09-10T20:31:43.882772+0000","last_fullsized":"2024-09-10T20:31:43.882772+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16002,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885008+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:43.885008+0000","last_peered":"2024-09-10T20:31:43.885008+0000","last_clean":"2024-09-10T20:31:43.885008+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:43.885008+0000","last_undegraded":"2024-09-10T20:31:43.885008+0000","last_fullsized":"2024-09-10T20:31:43.885008+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12413,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885200+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:43.885200+0000","last_peered":"2024-09-10T20:31:43.885200+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:43.885200+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":13521,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:45.777437+0000","last_change":"2024-09-10T20:31:42.530238+0000","last_active":"2024-09-10T20:31:45.777437+0000","last_peered":"2024-09-10T20:31:45.777437+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:45.777437+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.961301+0000","last_change":"2024-09-10T20:31:43.960747+0000","last_active":"2024-09-10T20:31:43.961301+0000","last_peered":"2024-09-10T20:31:43.961301+0000","last_clean":"2024-09-10T20:31:43.961301+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:43.961301+0000","last_undegraded":"2024-09-10T20:31:43.961301+0000","last_fullsized":"2024-09-10T20:31:43.961301+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16173,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883190+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:43.883190+0000","last_peered":"2024-09-10T20:31:43.883190+0000","last_clean":"2024-09-10T20:31:43.883190+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:43.883190+0000","last_undegraded":"2024-09-10T20:31:43.883190+0000","last_fullsized":"2024-09-10T20:31:43.883190+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882965+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:43.882965+0000","last_peered":"2024-09-10T20:31:43.882965+0000","last_clean":"2024-09-10T20:31:43.882965+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:43.882965+0000","last_undegraded":"2024-09-10T20:31:43.882965+0000","last_fullsized":"2024-09-10T20:31:43.882965+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16249,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882885+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:43.882885+0000","last_peered":"2024-09-10T20:31:43.882885+0000","last_clean":"2024-09-10T20:31:43.882885+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:43.882885+0000","last_undegraded":"2024-09-10T20:31:43.882885+0000","last_fullsized":"2024-09-10T20:31:43.882885+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699924,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671916,"statfs":{"total":96636764160,"available":95920041984,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587500,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784340,"statfs":{"total":96636764160,"available":96035164160,"internally_reserved":0,"allocated":479232,"data_stored":198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:48.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:48.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12434, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.884857+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:43.884857+0000', 'last_peered': '2024-09-10T20:31:43.884857+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:43.884857+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12373, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.882606+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:43.882606+0000', 'last_peered': '2024-09-10T20:31:43.882606+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:43.882606+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12517, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.884761+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:43.884761+0000', 'last_peered': '2024-09-10T20:31:43.884761+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:43.884761+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12413, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.885200+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:43.885200+0000', 'last_peered': '2024-09-10T20:31:43.885200+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:43.885200+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:48.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 13521, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:45.777437+0000', 'last_change': '2024-09-10T20:31:42.530238+0000', 'last_active': '2024-09-10T20:31:45.777437+0000', 'last_peered': '2024-09-10T20:31:45.777437+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:45.777437+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1347, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 633, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:48.284 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:48.676 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8827/36 objects degraded (24519.444%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":797,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2755432448,"bytes_avail":383791624192,"bytes_total":386547056640,"degraded_objects":8827,"degraded_total":36,"degraded_ratio":245.19444444444446},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-09-10T20:31:48.149584+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:31:48.690 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:31:48.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:48.987 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:48.988 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:49.000 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-10T20:31:48.149530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690852,"kb_used_data":1800,"kb_used_omap":14,"kb_used_meta":2688945,"kb_avail":374796508,"statfs":{"total":386547056640,"available":383791624192,"internally_reserved":0,"allocated":1843200,"data_stored":720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2753480246},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001273"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12434,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884857+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:43.884857+0000","last_peered":"2024-09-10T20:31:43.884857+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:43.884857+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12414,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:43.883311+0000","last_peered":"2024-09-10T20:31:43.883311+0000","last_clean":"2024-09-10T20:31:43.883311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:43.883311+0000","last_undegraded":"2024-09-10T20:31:43.883311+0000","last_fullsized":"2024-09-10T20:31:43.883311+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883006+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:43.883006+0000","last_peered":"2024-09-10T20:31:43.883006+0000","last_clean":"2024-09-10T20:31:43.883006+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:43.883006+0000","last_undegraded":"2024-09-10T20:31:43.883006+0000","last_fullsized":"2024-09-10T20:31:43.883006+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12517,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884761+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:43.884761+0000","last_peered":"2024-09-10T20:31:43.884761+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:43.884761+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16074,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885128+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:43.885128+0000","last_peered":"2024-09-10T20:31:43.885128+0000","last_clean":"2024-09-10T20:31:43.885128+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:43.885128+0000","last_undegraded":"2024-09-10T20:31:43.885128+0000","last_fullsized":"2024-09-10T20:31:43.885128+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.881097+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:43.881097+0000","last_peered":"2024-09-10T20:31:43.881097+0000","last_clean":"2024-09-10T20:31:43.881097+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:43.881097+0000","last_undegraded":"2024-09-10T20:31:43.881097+0000","last_fullsized":"2024-09-10T20:31:43.881097+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12500,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882772+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:43.882772+0000","last_peered":"2024-09-10T20:31:43.882772+0000","last_clean":"2024-09-10T20:31:43.882772+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:43.882772+0000","last_undegraded":"2024-09-10T20:31:43.882772+0000","last_fullsized":"2024-09-10T20:31:43.882772+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16002,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885008+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:43.885008+0000","last_peered":"2024-09-10T20:31:43.885008+0000","last_clean":"2024-09-10T20:31:43.885008+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:43.885008+0000","last_undegraded":"2024-09-10T20:31:43.885008+0000","last_fullsized":"2024-09-10T20:31:43.885008+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12413,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885200+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:43.885200+0000","last_peered":"2024-09-10T20:31:43.885200+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:43.885200+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":13521,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:45.777437+0000","last_change":"2024-09-10T20:31:42.530238+0000","last_active":"2024-09-10T20:31:45.777437+0000","last_peered":"2024-09-10T20:31:45.777437+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:45.777437+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.961301+0000","last_change":"2024-09-10T20:31:43.960747+0000","last_active":"2024-09-10T20:31:43.961301+0000","last_peered":"2024-09-10T20:31:43.961301+0000","last_clean":"2024-09-10T20:31:43.961301+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:43.961301+0000","last_undegraded":"2024-09-10T20:31:43.961301+0000","last_fullsized":"2024-09-10T20:31:43.961301+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16173,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883190+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:43.883190+0000","last_peered":"2024-09-10T20:31:43.883190+0000","last_clean":"2024-09-10T20:31:43.883190+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:43.883190+0000","last_undegraded":"2024-09-10T20:31:43.883190+0000","last_fullsized":"2024-09-10T20:31:43.883190+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882965+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:43.882965+0000","last_peered":"2024-09-10T20:31:43.882965+0000","last_clean":"2024-09-10T20:31:43.882965+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:43.882965+0000","last_undegraded":"2024-09-10T20:31:43.882965+0000","last_fullsized":"2024-09-10T20:31:43.882965+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16249,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882885+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:43.882885+0000","last_peered":"2024-09-10T20:31:43.882885+0000","last_clean":"2024-09-10T20:31:43.882885+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:43.882885+0000","last_undegraded":"2024-09-10T20:31:43.882885+0000","last_fullsized":"2024-09-10T20:31:43.882885+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117355,"num_write_kb":383270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699924,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671916,"statfs":{"total":96636764160,"available":95920041984,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587500,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784340,"statfs":{"total":96636764160,"available":96035164160,"internally_reserved":0,"allocated":479232,"data_stored":198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:52.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:52.309 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:52.309 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:52.321 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-10T20:31:52.150298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3901,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695004,"kb_used_data":1800,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792356,"statfs":{"total":386547056640,"available":383787372544,"internally_reserved":0,"allocated":1843200,"data_stored":720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11829,"num_write_kb":-47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.274424"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12434,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884857+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:43.884857+0000","last_peered":"2024-09-10T20:31:43.884857+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:43.884857+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12414,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:43.883311+0000","last_peered":"2024-09-10T20:31:43.883311+0000","last_clean":"2024-09-10T20:31:43.883311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:43.883311+0000","last_undegraded":"2024-09-10T20:31:43.883311+0000","last_fullsized":"2024-09-10T20:31:43.883311+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883006+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:43.883006+0000","last_peered":"2024-09-10T20:31:43.883006+0000","last_clean":"2024-09-10T20:31:43.883006+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:43.883006+0000","last_undegraded":"2024-09-10T20:31:43.883006+0000","last_fullsized":"2024-09-10T20:31:43.883006+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12518,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885802+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:43.885802+0000","last_peered":"2024-09-10T20:31:43.885802+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:43.885802+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16074,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885128+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:43.885128+0000","last_peered":"2024-09-10T20:31:43.885128+0000","last_clean":"2024-09-10T20:31:43.885128+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:43.885128+0000","last_undegraded":"2024-09-10T20:31:43.885128+0000","last_fullsized":"2024-09-10T20:31:43.885128+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.881097+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:43.881097+0000","last_peered":"2024-09-10T20:31:43.881097+0000","last_clean":"2024-09-10T20:31:43.881097+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:43.881097+0000","last_undegraded":"2024-09-10T20:31:43.881097+0000","last_fullsized":"2024-09-10T20:31:43.881097+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12500,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882772+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:43.882772+0000","last_peered":"2024-09-10T20:31:43.882772+0000","last_clean":"2024-09-10T20:31:43.882772+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:43.882772+0000","last_undegraded":"2024-09-10T20:31:43.882772+0000","last_fullsized":"2024-09-10T20:31:43.882772+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16002,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885008+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:43.885008+0000","last_peered":"2024-09-10T20:31:43.885008+0000","last_clean":"2024-09-10T20:31:43.885008+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:43.885008+0000","last_undegraded":"2024-09-10T20:31:43.885008+0000","last_fullsized":"2024-09-10T20:31:43.885008+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12413,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885200+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:43.885200+0000","last_peered":"2024-09-10T20:31:43.885200+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:43.885200+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":15389,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:50.780525+0000","last_change":"2024-09-10T20:31:42.530238+0000","last_active":"2024-09-10T20:31:50.780525+0000","last_peered":"2024-09-10T20:31:50.780525+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:50.780525+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12046,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:44.966645+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:44.966645+0000","last_peered":"2024-09-10T20:31:44.966645+0000","last_clean":"2024-09-10T20:31:44.966645+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:44.966645+0000","last_undegraded":"2024-09-10T20:31:44.966645+0000","last_fullsized":"2024-09-10T20:31:44.966645+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16173,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883190+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:43.883190+0000","last_peered":"2024-09-10T20:31:43.883190+0000","last_clean":"2024-09-10T20:31:43.883190+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:43.883190+0000","last_undegraded":"2024-09-10T20:31:43.883190+0000","last_fullsized":"2024-09-10T20:31:43.883190+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882965+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:43.882965+0000","last_peered":"2024-09-10T20:31:43.882965+0000","last_clean":"2024-09-10T20:31:43.882965+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:43.882965+0000","last_undegraded":"2024-09-10T20:31:43.882965+0000","last_fullsized":"2024-09-10T20:31:43.882965+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16249,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882885+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:43.882885+0000","last_peered":"2024-09-10T20:31:43.882885+0000","last_clean":"2024-09-10T20:31:43.882885+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:43.882885+0000","last_undegraded":"2024-09-10T20:31:43.882885+0000","last_fullsized":"2024-09-10T20:31:43.882885+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3901,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699924,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671916,"statfs":{"total":96636764160,"available":95920041984,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591652,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780188,"statfs":{"total":96636764160,"available":96030912512,"internally_reserved":0,"allocated":479232,"data_stored":198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:52.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:52.657 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:52.658 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:52.670 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-10T20:31:52.150298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3901,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695004,"kb_used_data":1800,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792356,"statfs":{"total":386547056640,"available":383787372544,"internally_reserved":0,"allocated":1843200,"data_stored":720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11829,"num_write_kb":-47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.274424"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12434,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884857+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:43.884857+0000","last_peered":"2024-09-10T20:31:43.884857+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:43.884857+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12414,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:43.883311+0000","last_peered":"2024-09-10T20:31:43.883311+0000","last_clean":"2024-09-10T20:31:43.883311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:43.883311+0000","last_undegraded":"2024-09-10T20:31:43.883311+0000","last_fullsized":"2024-09-10T20:31:43.883311+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883006+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:43.883006+0000","last_peered":"2024-09-10T20:31:43.883006+0000","last_clean":"2024-09-10T20:31:43.883006+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:43.883006+0000","last_undegraded":"2024-09-10T20:31:43.883006+0000","last_fullsized":"2024-09-10T20:31:43.883006+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12518,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885802+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:43.885802+0000","last_peered":"2024-09-10T20:31:43.885802+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:43.885802+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16074,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885128+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:43.885128+0000","last_peered":"2024-09-10T20:31:43.885128+0000","last_clean":"2024-09-10T20:31:43.885128+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:43.885128+0000","last_undegraded":"2024-09-10T20:31:43.885128+0000","last_fullsized":"2024-09-10T20:31:43.885128+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.881097+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:43.881097+0000","last_peered":"2024-09-10T20:31:43.881097+0000","last_clean":"2024-09-10T20:31:43.881097+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:43.881097+0000","last_undegraded":"2024-09-10T20:31:43.881097+0000","last_fullsized":"2024-09-10T20:31:43.881097+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12500,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882772+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:43.882772+0000","last_peered":"2024-09-10T20:31:43.882772+0000","last_clean":"2024-09-10T20:31:43.882772+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:43.882772+0000","last_undegraded":"2024-09-10T20:31:43.882772+0000","last_fullsized":"2024-09-10T20:31:43.882772+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16002,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885008+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:43.885008+0000","last_peered":"2024-09-10T20:31:43.885008+0000","last_clean":"2024-09-10T20:31:43.885008+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:43.885008+0000","last_undegraded":"2024-09-10T20:31:43.885008+0000","last_fullsized":"2024-09-10T20:31:43.885008+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12413,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885200+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:43.885200+0000","last_peered":"2024-09-10T20:31:43.885200+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:43.885200+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":15389,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:50.780525+0000","last_change":"2024-09-10T20:31:42.530238+0000","last_active":"2024-09-10T20:31:50.780525+0000","last_peered":"2024-09-10T20:31:50.780525+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:50.780525+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12046,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:44.966645+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:44.966645+0000","last_peered":"2024-09-10T20:31:44.966645+0000","last_clean":"2024-09-10T20:31:44.966645+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:44.966645+0000","last_undegraded":"2024-09-10T20:31:44.966645+0000","last_fullsized":"2024-09-10T20:31:44.966645+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16173,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883190+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:43.883190+0000","last_peered":"2024-09-10T20:31:43.883190+0000","last_clean":"2024-09-10T20:31:43.883190+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:43.883190+0000","last_undegraded":"2024-09-10T20:31:43.883190+0000","last_fullsized":"2024-09-10T20:31:43.883190+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882965+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:43.882965+0000","last_peered":"2024-09-10T20:31:43.882965+0000","last_clean":"2024-09-10T20:31:43.882965+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:43.882965+0000","last_undegraded":"2024-09-10T20:31:43.882965+0000","last_fullsized":"2024-09-10T20:31:43.882965+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16249,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882885+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:43.882885+0000","last_peered":"2024-09-10T20:31:43.882885+0000","last_clean":"2024-09-10T20:31:43.882885+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:43.882885+0000","last_undegraded":"2024-09-10T20:31:43.882885+0000","last_fullsized":"2024-09-10T20:31:43.882885+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3901,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699924,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671916,"statfs":{"total":96636764160,"available":95920041984,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591652,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780188,"statfs":{"total":96636764160,"available":96030912512,"internally_reserved":0,"allocated":479232,"data_stored":198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:52.670 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12434, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.884857+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:43.884857+0000', 'last_peered': '2024-09-10T20:31:43.884857+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:43.884857+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12373, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.882606+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:43.882606+0000', 'last_peered': '2024-09-10T20:31:43.882606+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:43.882606+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12518, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.885802+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:43.885802+0000', 'last_peered': '2024-09-10T20:31:43.885802+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:43.885802+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12413, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.885200+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:43.885200+0000', 'last_peered': '2024-09-10T20:31:43.885200+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:43.885200+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-10T20:31:52.671 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "44'12164", 'reported_seq': 15389, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:50.780525+0000', 'last_change': '2024-09-10T20:31:42.530238+0000', 'last_active': '2024-09-10T20:31:50.780525+0000', 'last_peered': '2024-09-10T20:31:50.780525+0000', 'last_clean': '2024-09-10T20:18:59.041285+0000', 'last_became_active': '2024-09-10T20:20:16.137833+0000', 'last_became_peered': '2024-09-10T20:20:16.137833+0000', 'last_unstale': '2024-09-10T20:31:50.780525+0000', 'last_undegraded': '2024-09-10T20:20:15.517601+0000', 'last_fullsized': '2024-09-10T20:20:15.455701+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 413, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 48655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1567, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -412}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:52.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:53.049 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:53.049 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7893/32 objects degraded (24665.625%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":801,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":53888,"bytes_used":2759684096,"bytes_avail":383787372544,"bytes_total":386547056640,"degraded_objects":7893,"degraded_total":32,"degraded_ratio":246.65625},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-09-10T20:31:48.149584+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:31:53.062 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:31:53.062 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:53.370 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:53.371 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-10T20:31:52.150298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3901,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695004,"kb_used_data":1800,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792356,"statfs":{"total":386547056640,"available":383787372544,"internally_reserved":0,"allocated":1843200,"data_stored":720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11829,"num_write_kb":-47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.274424"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12434,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.884857+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:43.884857+0000","last_peered":"2024-09-10T20:31:43.884857+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:43.884857+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12414,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883311+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:43.883311+0000","last_peered":"2024-09-10T20:31:43.883311+0000","last_clean":"2024-09-10T20:31:43.883311+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:43.883311+0000","last_undegraded":"2024-09-10T20:31:43.883311+0000","last_fullsized":"2024-09-10T20:31:43.883311+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883006+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:43.883006+0000","last_peered":"2024-09-10T20:31:43.883006+0000","last_clean":"2024-09-10T20:31:43.883006+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:43.883006+0000","last_undegraded":"2024-09-10T20:31:43.883006+0000","last_fullsized":"2024-09-10T20:31:43.883006+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12518,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885802+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:43.885802+0000","last_peered":"2024-09-10T20:31:43.885802+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:43.885802+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16074,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885128+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:43.885128+0000","last_peered":"2024-09-10T20:31:43.885128+0000","last_clean":"2024-09-10T20:31:43.885128+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:43.885128+0000","last_undegraded":"2024-09-10T20:31:43.885128+0000","last_fullsized":"2024-09-10T20:31:43.885128+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.881097+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:43.881097+0000","last_peered":"2024-09-10T20:31:43.881097+0000","last_clean":"2024-09-10T20:31:43.881097+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:43.881097+0000","last_undegraded":"2024-09-10T20:31:43.881097+0000","last_fullsized":"2024-09-10T20:31:43.881097+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12500,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882772+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:43.882772+0000","last_peered":"2024-09-10T20:31:43.882772+0000","last_clean":"2024-09-10T20:31:43.882772+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:43.882772+0000","last_undegraded":"2024-09-10T20:31:43.882772+0000","last_fullsized":"2024-09-10T20:31:43.882772+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16002,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.885008+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:43.885008+0000","last_peered":"2024-09-10T20:31:43.885008+0000","last_clean":"2024-09-10T20:31:43.885008+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:43.885008+0000","last_undegraded":"2024-09-10T20:31:43.885008+0000","last_fullsized":"2024-09-10T20:31:43.885008+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12413,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.885200+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:43.885200+0000","last_peered":"2024-09-10T20:31:43.885200+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:43.885200+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":15389,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:50.780525+0000","last_change":"2024-09-10T20:31:42.530238+0000","last_active":"2024-09-10T20:31:50.780525+0000","last_peered":"2024-09-10T20:31:50.780525+0000","last_clean":"2024-09-10T20:18:59.041285+0000","last_became_active":"2024-09-10T20:20:16.137833+0000","last_became_peered":"2024-09-10T20:20:16.137833+0000","last_unstale":"2024-09-10T20:31:50.780525+0000","last_undegraded":"2024-09-10T20:20:15.517601+0000","last_fullsized":"2024-09-10T20:20:15.455701+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12046,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:44.966645+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:44.966645+0000","last_peered":"2024-09-10T20:31:44.966645+0000","last_clean":"2024-09-10T20:31:44.966645+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:44.966645+0000","last_undegraded":"2024-09-10T20:31:44.966645+0000","last_fullsized":"2024-09-10T20:31:44.966645+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16173,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883190+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:43.883190+0000","last_peered":"2024-09-10T20:31:43.883190+0000","last_clean":"2024-09-10T20:31:43.883190+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:43.883190+0000","last_undegraded":"2024-09-10T20:31:43.883190+0000","last_fullsized":"2024-09-10T20:31:43.883190+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882965+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:43.882965+0000","last_peered":"2024-09-10T20:31:43.882965+0000","last_clean":"2024-09-10T20:31:43.882965+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:43.882965+0000","last_undegraded":"2024-09-10T20:31:43.882965+0000","last_fullsized":"2024-09-10T20:31:43.882965+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16249,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.882885+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:43.882885+0000","last_peered":"2024-09-10T20:31:43.882885+0000","last_clean":"2024-09-10T20:31:43.882885+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:43.882885+0000","last_undegraded":"2024-09-10T20:31:43.882885+0000","last_fullsized":"2024-09-10T20:31:43.882885+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3901,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699924,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671916,"statfs":{"total":96636764160,"available":95920041984,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591652,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780188,"statfs":{"total":96636764160,"available":96030912512,"internally_reserved":0,"allocated":479232,"data_stored":198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699856,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671984,"statfs":{"total":96636764160,"available":95920111616,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:56.386 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:56.692 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:56.693 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-10T20:31:56.150956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703348,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374784012,"statfs":{"total":386547056640,"available":383778828288,"internally_reserved":0,"allocated":1871872,"data_stored":728747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11829,"num_write_kb":-47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-560,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16112,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-10T20:31:53.415950+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:53.415950+0000","last_peered":"2024-09-10T20:31:53.415950+0000","last_clean":"2024-09-10T20:31:53.415950+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:53.415950+0000","last_undegraded":"2024-09-10T20:31:53.415950+0000","last_fullsized":"2024-09-10T20:31:53.415950+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12487,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.504762+0000","last_change":"2024-09-10T20:31:54.504180+0000","last_active":"2024-09-10T20:31:54.504762+0000","last_peered":"2024-09-10T20:31:54.504762+0000","last_clean":"2024-09-10T20:31:54.504762+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:54.504762+0000","last_undegraded":"2024-09-10T20:31:54.504762+0000","last_fullsized":"2024-09-10T20:31:54.504762+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599976,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771864,"statfs":{"total":96636764160,"available":96022388736,"internally_reserved":0,"allocated":483328,"data_stored":197552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699864,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671976,"statfs":{"total":96636764160,"available":95920103424,"internally_reserved":0,"allocated":462848,"data_stored":177072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:56.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:57.032 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:57.032 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:57.044 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-10T20:31:56.150956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703348,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374784012,"statfs":{"total":386547056640,"available":383778828288,"internally_reserved":0,"allocated":1871872,"data_stored":728747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11829,"num_write_kb":-47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-560,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16112,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-10T20:31:53.415950+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:53.415950+0000","last_peered":"2024-09-10T20:31:53.415950+0000","last_clean":"2024-09-10T20:31:53.415950+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:53.415950+0000","last_undegraded":"2024-09-10T20:31:53.415950+0000","last_fullsized":"2024-09-10T20:31:53.415950+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12487,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.504762+0000","last_change":"2024-09-10T20:31:54.504180+0000","last_active":"2024-09-10T20:31:54.504762+0000","last_peered":"2024-09-10T20:31:54.504762+0000","last_clean":"2024-09-10T20:31:54.504762+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:54.504762+0000","last_undegraded":"2024-09-10T20:31:54.504762+0000","last_fullsized":"2024-09-10T20:31:54.504762+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599976,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771864,"statfs":{"total":96636764160,"available":96022388736,"internally_reserved":0,"allocated":483328,"data_stored":197552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699864,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671976,"statfs":{"total":96636764160,"available":95920103424,"internally_reserved":0,"allocated":462848,"data_stored":177072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:31:57.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:31:57.045 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12438, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432325+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:54.432325+0000', 'last_peered': '2024-09-10T20:31:54.432325+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:54.432325+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:57.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:31:57.045 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 12373, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:43.882606+0000', 'last_change': '2024-09-10T20:20:16.151790+0000', 'last_active': '2024-09-10T20:31:43.882606+0000', 'last_peered': '2024-09-10T20:31:43.882606+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:43.882606+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:31:57.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:31:57.046 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12522, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432120+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:54.432120+0000', 'last_peered': '2024-09-10T20:31:54.432120+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:54.432120+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:57.046 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:31:57.046 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12417, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432261+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:54.432261+0000', 'last_peered': '2024-09-10T20:31:54.432261+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:54.432261+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:31:57.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:31:57.428 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:57.429 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7480/32 objects degraded (23375.000%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":806,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":53888,"bytes_used":2768228352,"bytes_avail":383778828288,"bytes_total":386547056640,"degraded_objects":7480,"degraded_total":32,"degraded_ratio":233.75},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-09-10T20:31:48.149584+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:31:57.442 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:31:57.443 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:31:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:31:57.746 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:31:57.757 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-10T20:31:56.150956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703348,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374784012,"statfs":{"total":386547056640,"available":383778828288,"internally_reserved":0,"allocated":1871872,"data_stored":728747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11829,"num_write_kb":-47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-560,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":12373,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:43.882606+0000","last_change":"2024-09-10T20:20:16.151790+0000","last_active":"2024-09-10T20:31:43.882606+0000","last_peered":"2024-09-10T20:31:43.882606+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:43.882606+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16112,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-10T20:31:53.415950+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:53.415950+0000","last_peered":"2024-09-10T20:31:53.415950+0000","last_clean":"2024-09-10T20:31:53.415950+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:53.415950+0000","last_undegraded":"2024-09-10T20:31:53.415950+0000","last_fullsized":"2024-09-10T20:31:53.415950+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16143,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:31:43.883240+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:43.883240+0000","last_peered":"2024-09-10T20:31:43.883240+0000","last_clean":"2024-09-10T20:31:43.883240+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:43.883240+0000","last_undegraded":"2024-09-10T20:31:43.883240+0000","last_fullsized":"2024-09-10T20:31:43.883240+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12487,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.504762+0000","last_change":"2024-09-10T20:31:54.504180+0000","last_active":"2024-09-10T20:31:54.504762+0000","last_peered":"2024-09-10T20:31:54.504762+0000","last_clean":"2024-09-10T20:31:54.504762+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:54.504762+0000","last_undegraded":"2024-09-10T20:31:54.504762+0000","last_fullsized":"2024-09-10T20:31:54.504762+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105526,"num_write_kb":335951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703572,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668268,"statfs":{"total":96636764160,"available":95916306432,"internally_reserved":0,"allocated":454656,"data_stored":173856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599976,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771864,"statfs":{"total":96636764160,"available":96022388736,"internally_reserved":0,"allocated":483328,"data_stored":197552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699864,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671976,"statfs":{"total":96636764160,"available":95920103424,"internally_reserved":0,"allocated":462848,"data_stored":177072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:00.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:01.078 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:01.078 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:01.090 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-10T20:32:00.151659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695060,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792300,"statfs":{"total":386547056640,"available":383787315200,"internally_reserved":0,"allocated":1904640,"data_stored":741548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12163,"num_write_kb":-48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-748,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001361"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":14016,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:57.376767+0000","last_change":"2024-09-10T20:31:52.461602+0000","last_active":"2024-09-10T20:31:57.376767+0000","last_peered":"2024-09-10T20:31:57.376767+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:57.376767+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16114,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.427658+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:54.427658+0000","last_peered":"2024-09-10T20:31:54.427658+0000","last_clean":"2024-09-10T20:31:54.427658+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:54.427658+0000","last_undegraded":"2024-09-10T20:31:54.427658+0000","last_fullsized":"2024-09-10T20:31:54.427658+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16147,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.428077+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:54.428077+0000","last_peered":"2024-09-10T20:31:54.428077+0000","last_clean":"2024-09-10T20:31:54.428077+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:54.428077+0000","last_undegraded":"2024-09-10T20:31:54.428077+0000","last_fullsized":"2024-09-10T20:31:54.428077+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12495,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:55.067744+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:31:55.067744+0000","last_peered":"2024-09-10T20:31:55.067744+0000","last_clean":"2024-09-10T20:31:55.067744+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:55.067744+0000","last_undegraded":"2024-09-10T20:31:55.067744+0000","last_fullsized":"2024-09-10T20:31:55.067744+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703584,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668256,"statfs":{"total":96636764160,"available":95916294144,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591664,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780176,"statfs":{"total":96636764160,"available":96030900224,"internally_reserved":0,"allocated":491520,"data_stored":200747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:01.091 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:01.389 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:01.402 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-10T20:32:00.151659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695060,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792300,"statfs":{"total":386547056640,"available":383787315200,"internally_reserved":0,"allocated":1904640,"data_stored":741548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12163,"num_write_kb":-48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-748,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001361"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":14016,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:57.376767+0000","last_change":"2024-09-10T20:31:52.461602+0000","last_active":"2024-09-10T20:31:57.376767+0000","last_peered":"2024-09-10T20:31:57.376767+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:57.376767+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16114,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.427658+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:54.427658+0000","last_peered":"2024-09-10T20:31:54.427658+0000","last_clean":"2024-09-10T20:31:54.427658+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:54.427658+0000","last_undegraded":"2024-09-10T20:31:54.427658+0000","last_fullsized":"2024-09-10T20:31:54.427658+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16147,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.428077+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:54.428077+0000","last_peered":"2024-09-10T20:31:54.428077+0000","last_clean":"2024-09-10T20:31:54.428077+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:54.428077+0000","last_undegraded":"2024-09-10T20:31:54.428077+0000","last_fullsized":"2024-09-10T20:31:54.428077+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12495,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:55.067744+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:31:55.067744+0000","last_peered":"2024-09-10T20:31:55.067744+0000","last_clean":"2024-09-10T20:31:55.067744+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:55.067744+0000","last_undegraded":"2024-09-10T20:31:55.067744+0000","last_fullsized":"2024-09-10T20:31:55.067744+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703584,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668256,"statfs":{"total":96636764160,"available":95916294144,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591664,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780176,"statfs":{"total":96636764160,"available":96030900224,"internally_reserved":0,"allocated":491520,"data_stored":200747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12438, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432325+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:54.432325+0000', 'last_peered': '2024-09-10T20:31:54.432325+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:54.432325+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 14016, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:57.376767+0000', 'last_change': '2024-09-10T20:31:52.461602+0000', 'last_active': '2024-09-10T20:31:57.376767+0000', 'last_peered': '2024-09-10T20:31:57.376767+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:31:57.376767+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1090, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 874, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1089}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12522, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432120+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:54.432120+0000', 'last_peered': '2024-09-10T20:31:54.432120+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:54.432120+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:32:01.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12417, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432261+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:54.432261+0000', 'last_peered': '2024-09-10T20:31:54.432261+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:54.432261+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:01.404 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:01.806 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:01.806 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6661/28 objects degraded (23789.286%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":810,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47152,"bytes_used":2759741440,"bytes_avail":383787315200,"bytes_total":386547056640,"degraded_objects":6661,"degraded_total":28,"degraded_ratio":237.89285714285714},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-09-10T20:31:48.149584+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:32:01.819 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:32:01.819 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:02.122 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:02.122 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:02.134 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-10T20:32:00.151659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695060,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792300,"statfs":{"total":386547056640,"available":383787315200,"internally_reserved":0,"allocated":1904640,"data_stored":741548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12163,"num_write_kb":-48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-748,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001361"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":14016,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:57.376767+0000","last_change":"2024-09-10T20:31:52.461602+0000","last_active":"2024-09-10T20:31:57.376767+0000","last_peered":"2024-09-10T20:31:57.376767+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:31:57.376767+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16114,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.427658+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:54.427658+0000","last_peered":"2024-09-10T20:31:54.427658+0000","last_clean":"2024-09-10T20:31:54.427658+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:54.427658+0000","last_undegraded":"2024-09-10T20:31:54.427658+0000","last_fullsized":"2024-09-10T20:31:54.427658+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16147,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.428077+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:54.428077+0000","last_peered":"2024-09-10T20:31:54.428077+0000","last_clean":"2024-09-10T20:31:54.428077+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:54.428077+0000","last_undegraded":"2024-09-10T20:31:54.428077+0000","last_fullsized":"2024-09-10T20:31:54.428077+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12495,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:55.067744+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:31:55.067744+0000","last_peered":"2024-09-10T20:31:55.067744+0000","last_clean":"2024-09-10T20:31:55.067744+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:55.067744+0000","last_undegraded":"2024-09-10T20:31:55.067744+0000","last_fullsized":"2024-09-10T20:31:55.067744+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703584,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668256,"statfs":{"total":96636764160,"available":95916294144,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591664,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780176,"statfs":{"total":96636764160,"available":96030900224,"internally_reserved":0,"allocated":491520,"data_stored":200747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:05.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:05.435 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:05.447 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-10T20:32:04.152349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695060,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792300,"statfs":{"total":386547056640,"available":383787315200,"internally_reserved":0,"allocated":1904640,"data_stored":741548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12163,"num_write_kb":-48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1790,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001392"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16197,"reported_epoch":56,"state":"active+undersized+remapped","last_fresh":"2024-09-10T20:32:02.342194+0000","last_change":"2024-09-10T20:32:02.342407+0000","last_active":"2024-09-10T20:32:02.342194+0000","last_peered":"2024-09-10T20:32:02.342194+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:32:02.342194+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","1(1)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16114,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.427658+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:54.427658+0000","last_peered":"2024-09-10T20:31:54.427658+0000","last_clean":"2024-09-10T20:31:54.427658+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:54.427658+0000","last_undegraded":"2024-09-10T20:31:54.427658+0000","last_fullsized":"2024-09-10T20:31:54.427658+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16147,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.428077+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:54.428077+0000","last_peered":"2024-09-10T20:31:54.428077+0000","last_clean":"2024-09-10T20:31:54.428077+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:54.428077+0000","last_undegraded":"2024-09-10T20:31:54.428077+0000","last_fullsized":"2024-09-10T20:31:54.428077+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12495,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:55.067744+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:31:55.067744+0000","last_peered":"2024-09-10T20:31:55.067744+0000","last_clean":"2024-09-10T20:31:55.067744+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:55.067744+0000","last_undegraded":"2024-09-10T20:31:55.067744+0000","last_fullsized":"2024-09-10T20:31:55.067744+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703584,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668256,"statfs":{"total":96636764160,"available":95916294144,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591664,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780176,"statfs":{"total":96636764160,"available":96030900224,"internally_reserved":0,"allocated":491520,"data_stored":200747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:05.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:05.779 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:05.779 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:05.792 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-10T20:32:04.152349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695060,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792300,"statfs":{"total":386547056640,"available":383787315200,"internally_reserved":0,"allocated":1904640,"data_stored":741548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12163,"num_write_kb":-48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1790,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001392"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12438,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432325+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:31:54.432325+0000","last_peered":"2024-09-10T20:31:54.432325+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:31:54.432325+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12418,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429859+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:31:54.429859+0000","last_peered":"2024-09-10T20:31:54.429859+0000","last_clean":"2024-09-10T20:31:54.429859+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:31:54.429859+0000","last_undegraded":"2024-09-10T20:31:54.429859+0000","last_fullsized":"2024-09-10T20:31:54.429859+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16197,"reported_epoch":56,"state":"active+undersized+remapped","last_fresh":"2024-09-10T20:32:02.342194+0000","last_change":"2024-09-10T20:32:02.342407+0000","last_active":"2024-09-10T20:32:02.342194+0000","last_peered":"2024-09-10T20:32:02.342194+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:32:02.342194+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","1(1)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16610,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429323+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:31:54.429323+0000","last_peered":"2024-09-10T20:31:54.429323+0000","last_clean":"2024-09-10T20:31:54.429323+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:31:54.429323+0000","last_undegraded":"2024-09-10T20:31:54.429323+0000","last_fullsized":"2024-09-10T20:31:54.429323+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":12522,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432120+0000","last_change":"2024-09-10T20:20:16.143444+0000","last_active":"2024-09-10T20:31:54.432120+0000","last_peered":"2024-09-10T20:31:54.432120+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:31:54.432120+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16078,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432387+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:31:54.432387+0000","last_peered":"2024-09-10T20:31:54.432387+0000","last_clean":"2024-09-10T20:31:54.432387+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:31:54.432387+0000","last_undegraded":"2024-09-10T20:31:54.432387+0000","last_fullsized":"2024-09-10T20:31:54.432387+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16114,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.427658+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:31:54.427658+0000","last_peered":"2024-09-10T20:31:54.427658+0000","last_clean":"2024-09-10T20:31:54.427658+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:31:54.427658+0000","last_undegraded":"2024-09-10T20:31:54.427658+0000","last_fullsized":"2024-09-10T20:31:54.427658+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12504,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429532+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:31:54.429532+0000","last_peered":"2024-09-10T20:31:54.429532+0000","last_clean":"2024-09-10T20:31:54.429532+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:31:54.429532+0000","last_undegraded":"2024-09-10T20:31:54.429532+0000","last_fullsized":"2024-09-10T20:31:54.429532+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16006,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.432208+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:31:54.432208+0000","last_peered":"2024-09-10T20:31:54.432208+0000","last_clean":"2024-09-10T20:31:54.432208+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:31:54.432208+0000","last_undegraded":"2024-09-10T20:31:54.432208+0000","last_fullsized":"2024-09-10T20:31:54.432208+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16147,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.428077+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:54.428077+0000","last_peered":"2024-09-10T20:31:54.428077+0000","last_clean":"2024-09-10T20:31:54.428077+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:54.428077+0000","last_undegraded":"2024-09-10T20:31:54.428077+0000","last_fullsized":"2024-09-10T20:31:54.428077+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12417,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:31:54.432261+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:31:54.432261+0000","last_peered":"2024-09-10T20:31:54.432261+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:31:54.432261+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12495,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:55.067744+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:31:55.067744+0000","last_peered":"2024-09-10T20:31:55.067744+0000","last_clean":"2024-09-10T20:31:55.067744+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:31:55.067744+0000","last_undegraded":"2024-09-10T20:31:55.067744+0000","last_fullsized":"2024-09-10T20:31:55.067744+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12050,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429396+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:31:54.429396+0000","last_peered":"2024-09-10T20:31:54.429396+0000","last_clean":"2024-09-10T20:31:54.429396+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:31:54.429396+0000","last_undegraded":"2024-09-10T20:31:54.429396+0000","last_fullsized":"2024-09-10T20:31:54.429396+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16177,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429591+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:31:54.429591+0000","last_peered":"2024-09-10T20:31:54.429591+0000","last_clean":"2024-09-10T20:31:54.429591+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:31:54.429591+0000","last_undegraded":"2024-09-10T20:31:54.429591+0000","last_fullsized":"2024-09-10T20:31:54.429591+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12701,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.430273+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:31:54.430273+0000","last_peered":"2024-09-10T20:31:54.430273+0000","last_clean":"2024-09-10T20:31:54.430273+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:31:54.430273+0000","last_undegraded":"2024-09-10T20:31:54.430273+0000","last_fullsized":"2024-09-10T20:31:54.430273+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16253,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.429632+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:31:54.429632+0000","last_peered":"2024-09-10T20:31:54.429632+0000","last_clean":"2024-09-10T20:31:54.429632+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:31:54.429632+0000","last_undegraded":"2024-09-10T20:31:54.429632+0000","last_fullsized":"2024-09-10T20:31:54.429632+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699936,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671904,"statfs":{"total":96636764160,"available":95920029696,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703584,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668256,"statfs":{"total":96636764160,"available":95916294144,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591664,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780176,"statfs":{"total":96636764160,"available":96030900224,"internally_reserved":0,"allocated":491520,"data_stored":200747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699876,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93671964,"statfs":{"total":96636764160,"available":95920091136,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716173007},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12438, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432325+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:31:54.432325+0000', 'last_peered': '2024-09-10T20:31:54.432325+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:31:54.432325+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "44'12140", 'reported_seq': 16197, 'reported_epoch': 56, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-10T20:32:02.342194+0000', 'last_change': '2024-09-10T20:32:02.342407+0000', 'last_active': '2024-09-10T20:32:02.342194+0000', 'last_peered': '2024-09-10T20:32:02.342194+0000', 'last_clean': '2024-09-10T20:19:06.115614+0000', 'last_became_active': '2024-09-10T20:20:16.145389+0000', 'last_became_peered': '2024-09-10T20:20:16.145389+0000', 'last_unstale': '2024-09-10T20:32:02.342194+0000', 'last_undegraded': '2024-09-10T20:20:15.512914+0000', 'last_fullsized': '2024-09-10T20:20:15.494776+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12139, 'num_write_kb': 48551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1964, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '1(1)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 12522, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432120+0000', 'last_change': '2024-09-10T20:20:16.143444+0000', 'last_active': '2024-09-10T20:31:54.432120+0000', 'last_peered': '2024-09-10T20:31:54.432120+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:31:54.432120+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1857, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:32:05.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12417, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:31:54.432261+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:31:54.432261+0000', 'last_peered': '2024-09-10T20:31:54.432261+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:31:54.432261+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:05.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:06.183 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:06.183 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5571/28 objects degraded (19896.429%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":814,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47152,"bytes_used":2759741440,"bytes_avail":383787315200,"bytes_total":386547056640,"degraded_objects":5571,"degraded_total":28,"degraded_ratio":198.96428571428572,"recovering_objects_per_sec":223,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1790,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-09-10T20:31:48.149584+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:32:06.196 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:32:06.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:06.497 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:06.498 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:06.509 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-10T20:32:06.152639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707584,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779776,"statfs":{"total":386547056640,"available":383774490624,"internally_reserved":0,"allocated":1945600,"data_stored":752367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12163,"num_write_kb":-48655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1517,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001282"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12442,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.458272+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:03.458272+0000","last_peered":"2024-09-10T20:32:03.458272+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:03.458272+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12422,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457636+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:03.457636+0000","last_peered":"2024-09-10T20:32:03.457636+0000","last_clean":"2024-09-10T20:32:03.457636+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:03.457636+0000","last_undegraded":"2024-09-10T20:32:03.457636+0000","last_fullsized":"2024-09-10T20:32:03.457636+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16197,"reported_epoch":56,"state":"active+undersized+remapped","last_fresh":"2024-09-10T20:32:02.342194+0000","last_change":"2024-09-10T20:32:02.342407+0000","last_active":"2024-09-10T20:32:02.342194+0000","last_peered":"2024-09-10T20:32:02.342194+0000","last_clean":"2024-09-10T20:19:06.115614+0000","last_became_active":"2024-09-10T20:20:16.145389+0000","last_became_peered":"2024-09-10T20:20:16.145389+0000","last_unstale":"2024-09-10T20:32:02.342194+0000","last_undegraded":"2024-09-10T20:20:15.512914+0000","last_fullsized":"2024-09-10T20:20:15.494776+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12139,"num_write_kb":48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","1(1)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16614,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457171+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:03.457171+0000","last_peered":"2024-09-10T20:32:03.457171+0000","last_clean":"2024-09-10T20:32:03.457171+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:03.457171+0000","last_undegraded":"2024-09-10T20:32:03.457171+0000","last_fullsized":"2024-09-10T20:32:03.457171+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":13619,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:05.779506+0000","last_change":"2024-09-10T20:32:02.343201+0000","last_active":"2024-09-10T20:32:05.779506+0000","last_peered":"2024-09-10T20:32:05.779506+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:32:05.779506+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16082,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457813+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:03.457813+0000","last_peered":"2024-09-10T20:32:03.457813+0000","last_clean":"2024-09-10T20:32:03.457813+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:03.457813+0000","last_undegraded":"2024-09-10T20:32:03.457813+0000","last_fullsized":"2024-09-10T20:32:03.457813+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16118,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455567+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:03.455567+0000","last_peered":"2024-09-10T20:32:03.455567+0000","last_clean":"2024-09-10T20:32:03.455567+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:03.455567+0000","last_undegraded":"2024-09-10T20:32:03.455567+0000","last_fullsized":"2024-09-10T20:32:03.455567+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12508,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457300+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:03.457300+0000","last_peered":"2024-09-10T20:32:03.457300+0000","last_clean":"2024-09-10T20:32:03.457300+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:03.457300+0000","last_undegraded":"2024-09-10T20:32:03.457300+0000","last_fullsized":"2024-09-10T20:32:03.457300+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457755+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:03.457755+0000","last_peered":"2024-09-10T20:32:03.457755+0000","last_clean":"2024-09-10T20:32:03.457755+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:03.457755+0000","last_undegraded":"2024-09-10T20:32:03.457755+0000","last_fullsized":"2024-09-10T20:32:03.457755+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16147,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:31:54.428077+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:31:54.428077+0000","last_peered":"2024-09-10T20:31:54.428077+0000","last_clean":"2024-09-10T20:31:54.428077+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:31:54.428077+0000","last_undegraded":"2024-09-10T20:31:54.428077+0000","last_fullsized":"2024-09-10T20:31:54.428077+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12421,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.457723+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:32:03.457723+0000","last_peered":"2024-09-10T20:32:03.457723+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:03.457723+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12499,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457681+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:03.457681+0000","last_peered":"2024-09-10T20:32:03.457681+0000","last_clean":"2024-09-10T20:32:03.457681+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:03.457681+0000","last_undegraded":"2024-09-10T20:32:03.457681+0000","last_fullsized":"2024-09-10T20:32:03.457681+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12054,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457177+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:03.457177+0000","last_peered":"2024-09-10T20:32:03.457177+0000","last_clean":"2024-09-10T20:32:03.457177+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:03.457177+0000","last_undegraded":"2024-09-10T20:32:03.457177+0000","last_fullsized":"2024-09-10T20:32:03.457177+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.461644+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:03.461644+0000","last_peered":"2024-09-10T20:32:03.461644+0000","last_clean":"2024-09-10T20:32:03.461644+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:03.461644+0000","last_undegraded":"2024-09-10T20:32:03.461644+0000","last_fullsized":"2024-09-10T20:32:03.461644+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12705,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457390+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:03.457390+0000","last_peered":"2024-09-10T20:32:03.457390+0000","last_clean":"2024-09-10T20:32:03.457390+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:03.457390+0000","last_undegraded":"2024-09-10T20:32:03.457390+0000","last_fullsized":"2024-09-10T20:32:03.457390+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16257,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457564+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:03.457564+0000","last_peered":"2024-09-10T20:32:03.457564+0000","last_clean":"2024-09-10T20:32:03.457564+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:03.457564+0000","last_undegraded":"2024-09-10T20:32:03.457564+0000","last_fullsized":"2024-09-10T20:32:03.457564+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93363,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699952,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671888,"statfs":{"total":96636764160,"available":95920013312,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703584,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668256,"statfs":{"total":96636764160,"available":95916294144,"internally_reserved":0,"allocated":471040,"data_stored":180267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599996,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771844,"statfs":{"total":96636764160,"available":96022368256,"internally_reserved":0,"allocated":499712,"data_stored":198892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704052,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667788,"statfs":{"total":96636764160,"available":95915814912,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:09.511 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:09.814 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:09.815 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:09.826 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-10T20:32:08.152968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2825,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707600,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779760,"statfs":{"total":386547056640,"available":383774474240,"internally_reserved":0,"allocated":1961984,"data_stored":758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12139,"num_write_kb":-48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12442,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.458272+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:03.458272+0000","last_peered":"2024-09-10T20:32:03.458272+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:03.458272+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12422,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457636+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:03.457636+0000","last_peered":"2024-09-10T20:32:03.457636+0000","last_clean":"2024-09-10T20:32:03.457636+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:03.457636+0000","last_undegraded":"2024-09-10T20:32:03.457636+0000","last_fullsized":"2024-09-10T20:32:03.457636+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16217,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:04.557049+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:04.557049+0000","last_peered":"2024-09-10T20:32:04.557049+0000","last_clean":"2024-09-10T20:32:04.557049+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:04.557049+0000","last_undegraded":"2024-09-10T20:32:04.557049+0000","last_fullsized":"2024-09-10T20:32:04.557049+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16614,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457171+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:03.457171+0000","last_peered":"2024-09-10T20:32:03.457171+0000","last_clean":"2024-09-10T20:32:03.457171+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:03.457171+0000","last_undegraded":"2024-09-10T20:32:03.457171+0000","last_fullsized":"2024-09-10T20:32:03.457171+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":13619,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:05.779506+0000","last_change":"2024-09-10T20:32:02.343201+0000","last_active":"2024-09-10T20:32:05.779506+0000","last_peered":"2024-09-10T20:32:05.779506+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:32:05.779506+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16082,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457813+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:03.457813+0000","last_peered":"2024-09-10T20:32:03.457813+0000","last_clean":"2024-09-10T20:32:03.457813+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:03.457813+0000","last_undegraded":"2024-09-10T20:32:03.457813+0000","last_fullsized":"2024-09-10T20:32:03.457813+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16118,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455567+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:03.455567+0000","last_peered":"2024-09-10T20:32:03.455567+0000","last_clean":"2024-09-10T20:32:03.455567+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:03.455567+0000","last_undegraded":"2024-09-10T20:32:03.455567+0000","last_fullsized":"2024-09-10T20:32:03.455567+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12508,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457300+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:03.457300+0000","last_peered":"2024-09-10T20:32:03.457300+0000","last_clean":"2024-09-10T20:32:03.457300+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:03.457300+0000","last_undegraded":"2024-09-10T20:32:03.457300+0000","last_fullsized":"2024-09-10T20:32:03.457300+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457755+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:03.457755+0000","last_peered":"2024-09-10T20:32:03.457755+0000","last_clean":"2024-09-10T20:32:03.457755+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:03.457755+0000","last_undegraded":"2024-09-10T20:32:03.457755+0000","last_fullsized":"2024-09-10T20:32:03.457755+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16151,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455848+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:03.455848+0000","last_peered":"2024-09-10T20:32:03.455848+0000","last_clean":"2024-09-10T20:32:03.455848+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:03.455848+0000","last_undegraded":"2024-09-10T20:32:03.455848+0000","last_fullsized":"2024-09-10T20:32:03.455848+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12421,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.457723+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:32:03.457723+0000","last_peered":"2024-09-10T20:32:03.457723+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:03.457723+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12499,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457681+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:03.457681+0000","last_peered":"2024-09-10T20:32:03.457681+0000","last_clean":"2024-09-10T20:32:03.457681+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:03.457681+0000","last_undegraded":"2024-09-10T20:32:03.457681+0000","last_fullsized":"2024-09-10T20:32:03.457681+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12054,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457177+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:03.457177+0000","last_peered":"2024-09-10T20:32:03.457177+0000","last_clean":"2024-09-10T20:32:03.457177+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:03.457177+0000","last_undegraded":"2024-09-10T20:32:03.457177+0000","last_fullsized":"2024-09-10T20:32:03.457177+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.461644+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:03.461644+0000","last_peered":"2024-09-10T20:32:03.461644+0000","last_clean":"2024-09-10T20:32:03.461644+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:03.461644+0000","last_undegraded":"2024-09-10T20:32:03.461644+0000","last_fullsized":"2024-09-10T20:32:03.461644+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12705,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457390+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:03.457390+0000","last_peered":"2024-09-10T20:32:03.457390+0000","last_clean":"2024-09-10T20:32:03.457390+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:03.457390+0000","last_undegraded":"2024-09-10T20:32:03.457390+0000","last_fullsized":"2024-09-10T20:32:03.457390+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16257,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457564+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:03.457564+0000","last_peered":"2024-09-10T20:32:03.457564+0000","last_clean":"2024-09-10T20:32:03.457564+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:03.457564+0000","last_undegraded":"2024-09-10T20:32:03.457564+0000","last_fullsized":"2024-09-10T20:32:03.457564+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2825,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699952,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671888,"statfs":{"total":96636764160,"available":95920013312,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703600,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668240,"statfs":{"total":96636764160,"available":95916277760,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599996,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771844,"statfs":{"total":96636764160,"available":96022368256,"internally_reserved":0,"allocated":499712,"data_stored":198892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704052,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667788,"statfs":{"total":96636764160,"available":95915814912,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:09.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:10.128 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:10.129 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:10.140 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-10T20:32:08.152968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2825,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707600,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779760,"statfs":{"total":386547056640,"available":383774474240,"internally_reserved":0,"allocated":1961984,"data_stored":758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12139,"num_write_kb":-48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12442,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.458272+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:03.458272+0000","last_peered":"2024-09-10T20:32:03.458272+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:03.458272+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12422,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457636+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:03.457636+0000","last_peered":"2024-09-10T20:32:03.457636+0000","last_clean":"2024-09-10T20:32:03.457636+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:03.457636+0000","last_undegraded":"2024-09-10T20:32:03.457636+0000","last_fullsized":"2024-09-10T20:32:03.457636+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16217,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:04.557049+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:04.557049+0000","last_peered":"2024-09-10T20:32:04.557049+0000","last_clean":"2024-09-10T20:32:04.557049+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:04.557049+0000","last_undegraded":"2024-09-10T20:32:04.557049+0000","last_fullsized":"2024-09-10T20:32:04.557049+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16614,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457171+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:03.457171+0000","last_peered":"2024-09-10T20:32:03.457171+0000","last_clean":"2024-09-10T20:32:03.457171+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:03.457171+0000","last_undegraded":"2024-09-10T20:32:03.457171+0000","last_fullsized":"2024-09-10T20:32:03.457171+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":13619,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:05.779506+0000","last_change":"2024-09-10T20:32:02.343201+0000","last_active":"2024-09-10T20:32:05.779506+0000","last_peered":"2024-09-10T20:32:05.779506+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:32:05.779506+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16082,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457813+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:03.457813+0000","last_peered":"2024-09-10T20:32:03.457813+0000","last_clean":"2024-09-10T20:32:03.457813+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:03.457813+0000","last_undegraded":"2024-09-10T20:32:03.457813+0000","last_fullsized":"2024-09-10T20:32:03.457813+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16118,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455567+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:03.455567+0000","last_peered":"2024-09-10T20:32:03.455567+0000","last_clean":"2024-09-10T20:32:03.455567+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:03.455567+0000","last_undegraded":"2024-09-10T20:32:03.455567+0000","last_fullsized":"2024-09-10T20:32:03.455567+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12508,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457300+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:03.457300+0000","last_peered":"2024-09-10T20:32:03.457300+0000","last_clean":"2024-09-10T20:32:03.457300+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:03.457300+0000","last_undegraded":"2024-09-10T20:32:03.457300+0000","last_fullsized":"2024-09-10T20:32:03.457300+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457755+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:03.457755+0000","last_peered":"2024-09-10T20:32:03.457755+0000","last_clean":"2024-09-10T20:32:03.457755+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:03.457755+0000","last_undegraded":"2024-09-10T20:32:03.457755+0000","last_fullsized":"2024-09-10T20:32:03.457755+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16151,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455848+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:03.455848+0000","last_peered":"2024-09-10T20:32:03.455848+0000","last_clean":"2024-09-10T20:32:03.455848+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:03.455848+0000","last_undegraded":"2024-09-10T20:32:03.455848+0000","last_fullsized":"2024-09-10T20:32:03.455848+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12421,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.457723+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:32:03.457723+0000","last_peered":"2024-09-10T20:32:03.457723+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:03.457723+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12499,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457681+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:03.457681+0000","last_peered":"2024-09-10T20:32:03.457681+0000","last_clean":"2024-09-10T20:32:03.457681+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:03.457681+0000","last_undegraded":"2024-09-10T20:32:03.457681+0000","last_fullsized":"2024-09-10T20:32:03.457681+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12054,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457177+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:03.457177+0000","last_peered":"2024-09-10T20:32:03.457177+0000","last_clean":"2024-09-10T20:32:03.457177+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:03.457177+0000","last_undegraded":"2024-09-10T20:32:03.457177+0000","last_fullsized":"2024-09-10T20:32:03.457177+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.461644+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:03.461644+0000","last_peered":"2024-09-10T20:32:03.461644+0000","last_clean":"2024-09-10T20:32:03.461644+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:03.461644+0000","last_undegraded":"2024-09-10T20:32:03.461644+0000","last_fullsized":"2024-09-10T20:32:03.461644+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12705,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457390+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:03.457390+0000","last_peered":"2024-09-10T20:32:03.457390+0000","last_clean":"2024-09-10T20:32:03.457390+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:03.457390+0000","last_undegraded":"2024-09-10T20:32:03.457390+0000","last_fullsized":"2024-09-10T20:32:03.457390+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16257,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457564+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:03.457564+0000","last_peered":"2024-09-10T20:32:03.457564+0000","last_clean":"2024-09-10T20:32:03.457564+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:03.457564+0000","last_undegraded":"2024-09-10T20:32:03.457564+0000","last_fullsized":"2024-09-10T20:32:03.457564+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2825,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699952,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671888,"statfs":{"total":96636764160,"available":95920013312,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703600,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668240,"statfs":{"total":96636764160,"available":95916277760,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599996,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771844,"statfs":{"total":96636764160,"available":96022368256,"internally_reserved":0,"allocated":499712,"data_stored":198892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704052,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667788,"statfs":{"total":96636764160,"available":95915814912,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:10.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:10.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12442, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:03.458272+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:32:03.458272+0000', 'last_peered': '2024-09-10T20:32:03.458272+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:32:03.458272+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:10.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:32:10.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 13619, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:05.779506+0000', 'last_change': '2024-09-10T20:32:02.343201+0000', 'last_active': '2024-09-10T20:32:05.779506+0000', 'last_peered': '2024-09-10T20:32:05.779506+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:32:05.779506+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 673, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:10.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:32:10.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12421, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:03.457723+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:32:03.457723+0000', 'last_peered': '2024-09-10T20:32:03.457723+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:32:03.457723+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:10.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:10.551 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:10.551 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5025/24 objects degraded (20937.500%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":819,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40416,"bytes_used":2764058624,"bytes_avail":383782998016,"bytes_total":386547056640,"degraded_objects":5025,"degraded_total":24,"degraded_ratio":209.375},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-09-10T20:31:48.149584+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:32:10.565 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:32:10.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:10.880 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:10.881 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:10.894 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-10T20:32:10.153343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2825,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699276,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374788084,"statfs":{"total":386547056640,"available":383782998016,"internally_reserved":0,"allocated":1961984,"data_stored":758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2761999926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12139,"num_write_kb":-48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001376"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12442,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.458272+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:03.458272+0000","last_peered":"2024-09-10T20:32:03.458272+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:03.458272+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12422,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457636+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:03.457636+0000","last_peered":"2024-09-10T20:32:03.457636+0000","last_clean":"2024-09-10T20:32:03.457636+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:03.457636+0000","last_undegraded":"2024-09-10T20:32:03.457636+0000","last_fullsized":"2024-09-10T20:32:03.457636+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16217,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:04.557049+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:04.557049+0000","last_peered":"2024-09-10T20:32:04.557049+0000","last_clean":"2024-09-10T20:32:04.557049+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:04.557049+0000","last_undegraded":"2024-09-10T20:32:04.557049+0000","last_fullsized":"2024-09-10T20:32:04.557049+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16614,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457171+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:03.457171+0000","last_peered":"2024-09-10T20:32:03.457171+0000","last_clean":"2024-09-10T20:32:03.457171+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:03.457171+0000","last_undegraded":"2024-09-10T20:32:03.457171+0000","last_fullsized":"2024-09-10T20:32:03.457171+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":13619,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:05.779506+0000","last_change":"2024-09-10T20:32:02.343201+0000","last_active":"2024-09-10T20:32:05.779506+0000","last_peered":"2024-09-10T20:32:05.779506+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:32:05.779506+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16082,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457813+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:03.457813+0000","last_peered":"2024-09-10T20:32:03.457813+0000","last_clean":"2024-09-10T20:32:03.457813+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:03.457813+0000","last_undegraded":"2024-09-10T20:32:03.457813+0000","last_fullsized":"2024-09-10T20:32:03.457813+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16118,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455567+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:03.455567+0000","last_peered":"2024-09-10T20:32:03.455567+0000","last_clean":"2024-09-10T20:32:03.455567+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:03.455567+0000","last_undegraded":"2024-09-10T20:32:03.455567+0000","last_fullsized":"2024-09-10T20:32:03.455567+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12508,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457300+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:03.457300+0000","last_peered":"2024-09-10T20:32:03.457300+0000","last_clean":"2024-09-10T20:32:03.457300+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:03.457300+0000","last_undegraded":"2024-09-10T20:32:03.457300+0000","last_fullsized":"2024-09-10T20:32:03.457300+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457755+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:03.457755+0000","last_peered":"2024-09-10T20:32:03.457755+0000","last_clean":"2024-09-10T20:32:03.457755+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:03.457755+0000","last_undegraded":"2024-09-10T20:32:03.457755+0000","last_fullsized":"2024-09-10T20:32:03.457755+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16151,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455848+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:03.455848+0000","last_peered":"2024-09-10T20:32:03.455848+0000","last_clean":"2024-09-10T20:32:03.455848+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:03.455848+0000","last_undegraded":"2024-09-10T20:32:03.455848+0000","last_fullsized":"2024-09-10T20:32:03.455848+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12421,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.457723+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:32:03.457723+0000","last_peered":"2024-09-10T20:32:03.457723+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:03.457723+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12499,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457681+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:03.457681+0000","last_peered":"2024-09-10T20:32:03.457681+0000","last_clean":"2024-09-10T20:32:03.457681+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:03.457681+0000","last_undegraded":"2024-09-10T20:32:03.457681+0000","last_fullsized":"2024-09-10T20:32:03.457681+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12054,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457177+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:03.457177+0000","last_peered":"2024-09-10T20:32:03.457177+0000","last_clean":"2024-09-10T20:32:03.457177+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:03.457177+0000","last_undegraded":"2024-09-10T20:32:03.457177+0000","last_fullsized":"2024-09-10T20:32:03.457177+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.461644+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:03.461644+0000","last_peered":"2024-09-10T20:32:03.461644+0000","last_clean":"2024-09-10T20:32:03.461644+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:03.461644+0000","last_undegraded":"2024-09-10T20:32:03.461644+0000","last_fullsized":"2024-09-10T20:32:03.461644+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12705,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457390+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:03.457390+0000","last_peered":"2024-09-10T20:32:03.457390+0000","last_clean":"2024-09-10T20:32:03.457390+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:03.457390+0000","last_undegraded":"2024-09-10T20:32:03.457390+0000","last_fullsized":"2024-09-10T20:32:03.457390+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16257,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457564+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:03.457564+0000","last_peered":"2024-09-10T20:32:03.457564+0000","last_clean":"2024-09-10T20:32:03.457564+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:03.457564+0000","last_undegraded":"2024-09-10T20:32:03.457564+0000","last_fullsized":"2024-09-10T20:32:03.457564+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2825,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699952,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671888,"statfs":{"total":96636764160,"available":95920013312,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703600,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668240,"statfs":{"total":96636764160,"available":95916277760,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591672,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780168,"statfs":{"total":96636764160,"available":96030892032,"internally_reserved":0,"allocated":499712,"data_stored":198892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704052,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667788,"statfs":{"total":96636764160,"available":95915814912,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:13.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:14.241 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:14.254 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-10T20:32:14.153977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699276,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374788084,"statfs":{"total":386547056640,"available":383782998016,"internally_reserved":0,"allocated":1961984,"data_stored":758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2761999926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12139,"num_write_kb":-48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-995,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001337"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12442,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.458272+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:03.458272+0000","last_peered":"2024-09-10T20:32:03.458272+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:03.458272+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12422,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457636+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:03.457636+0000","last_peered":"2024-09-10T20:32:03.457636+0000","last_clean":"2024-09-10T20:32:03.457636+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:03.457636+0000","last_undegraded":"2024-09-10T20:32:03.457636+0000","last_fullsized":"2024-09-10T20:32:03.457636+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16217,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:04.557049+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:04.557049+0000","last_peered":"2024-09-10T20:32:04.557049+0000","last_clean":"2024-09-10T20:32:04.557049+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:04.557049+0000","last_undegraded":"2024-09-10T20:32:04.557049+0000","last_fullsized":"2024-09-10T20:32:04.557049+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16614,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457171+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:03.457171+0000","last_peered":"2024-09-10T20:32:03.457171+0000","last_clean":"2024-09-10T20:32:03.457171+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:03.457171+0000","last_undegraded":"2024-09-10T20:32:03.457171+0000","last_fullsized":"2024-09-10T20:32:03.457171+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":15557,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:10.783527+0000","last_change":"2024-09-10T20:32:02.343201+0000","last_active":"2024-09-10T20:32:10.783527+0000","last_peered":"2024-09-10T20:32:10.783527+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:32:10.783527+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1642,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16082,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457813+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:03.457813+0000","last_peered":"2024-09-10T20:32:03.457813+0000","last_clean":"2024-09-10T20:32:03.457813+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:03.457813+0000","last_undegraded":"2024-09-10T20:32:03.457813+0000","last_fullsized":"2024-09-10T20:32:03.457813+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16118,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455567+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:03.455567+0000","last_peered":"2024-09-10T20:32:03.455567+0000","last_clean":"2024-09-10T20:32:03.455567+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:03.455567+0000","last_undegraded":"2024-09-10T20:32:03.455567+0000","last_fullsized":"2024-09-10T20:32:03.455567+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12508,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457300+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:03.457300+0000","last_peered":"2024-09-10T20:32:03.457300+0000","last_clean":"2024-09-10T20:32:03.457300+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:03.457300+0000","last_undegraded":"2024-09-10T20:32:03.457300+0000","last_fullsized":"2024-09-10T20:32:03.457300+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457755+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:03.457755+0000","last_peered":"2024-09-10T20:32:03.457755+0000","last_clean":"2024-09-10T20:32:03.457755+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:03.457755+0000","last_undegraded":"2024-09-10T20:32:03.457755+0000","last_fullsized":"2024-09-10T20:32:03.457755+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16151,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455848+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:03.455848+0000","last_peered":"2024-09-10T20:32:03.455848+0000","last_clean":"2024-09-10T20:32:03.455848+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:03.455848+0000","last_undegraded":"2024-09-10T20:32:03.455848+0000","last_fullsized":"2024-09-10T20:32:03.455848+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12421,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.457723+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:32:03.457723+0000","last_peered":"2024-09-10T20:32:03.457723+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:03.457723+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12499,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457681+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:03.457681+0000","last_peered":"2024-09-10T20:32:03.457681+0000","last_clean":"2024-09-10T20:32:03.457681+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:03.457681+0000","last_undegraded":"2024-09-10T20:32:03.457681+0000","last_fullsized":"2024-09-10T20:32:03.457681+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12054,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457177+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:03.457177+0000","last_peered":"2024-09-10T20:32:03.457177+0000","last_clean":"2024-09-10T20:32:03.457177+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:03.457177+0000","last_undegraded":"2024-09-10T20:32:03.457177+0000","last_fullsized":"2024-09-10T20:32:03.457177+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.461644+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:03.461644+0000","last_peered":"2024-09-10T20:32:03.461644+0000","last_clean":"2024-09-10T20:32:03.461644+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:03.461644+0000","last_undegraded":"2024-09-10T20:32:03.461644+0000","last_fullsized":"2024-09-10T20:32:03.461644+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12705,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457390+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:03.457390+0000","last_peered":"2024-09-10T20:32:03.457390+0000","last_clean":"2024-09-10T20:32:03.457390+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:03.457390+0000","last_undegraded":"2024-09-10T20:32:03.457390+0000","last_fullsized":"2024-09-10T20:32:03.457390+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16257,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457564+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:03.457564+0000","last_peered":"2024-09-10T20:32:03.457564+0000","last_clean":"2024-09-10T20:32:03.457564+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:03.457564+0000","last_undegraded":"2024-09-10T20:32:03.457564+0000","last_fullsized":"2024-09-10T20:32:03.457564+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699952,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671888,"statfs":{"total":96636764160,"available":95920013312,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703600,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668240,"statfs":{"total":96636764160,"available":95916277760,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591672,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780168,"statfs":{"total":96636764160,"available":96030892032,"internally_reserved":0,"allocated":499712,"data_stored":198892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704052,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667788,"statfs":{"total":96636764160,"available":95915814912,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:14.256 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:14.561 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:14.561 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:14.574 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-10T20:32:14.153977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699276,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374788084,"statfs":{"total":386547056640,"available":383782998016,"internally_reserved":0,"allocated":1961984,"data_stored":758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2761999926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12139,"num_write_kb":-48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-995,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001337"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12442,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.458272+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:03.458272+0000","last_peered":"2024-09-10T20:32:03.458272+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:03.458272+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12422,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457636+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:03.457636+0000","last_peered":"2024-09-10T20:32:03.457636+0000","last_clean":"2024-09-10T20:32:03.457636+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:03.457636+0000","last_undegraded":"2024-09-10T20:32:03.457636+0000","last_fullsized":"2024-09-10T20:32:03.457636+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16217,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:04.557049+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:04.557049+0000","last_peered":"2024-09-10T20:32:04.557049+0000","last_clean":"2024-09-10T20:32:04.557049+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:04.557049+0000","last_undegraded":"2024-09-10T20:32:04.557049+0000","last_fullsized":"2024-09-10T20:32:04.557049+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16614,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457171+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:03.457171+0000","last_peered":"2024-09-10T20:32:03.457171+0000","last_clean":"2024-09-10T20:32:03.457171+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:03.457171+0000","last_undegraded":"2024-09-10T20:32:03.457171+0000","last_fullsized":"2024-09-10T20:32:03.457171+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":15557,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:10.783527+0000","last_change":"2024-09-10T20:32:02.343201+0000","last_active":"2024-09-10T20:32:10.783527+0000","last_peered":"2024-09-10T20:32:10.783527+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:32:10.783527+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1642,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16082,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457813+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:03.457813+0000","last_peered":"2024-09-10T20:32:03.457813+0000","last_clean":"2024-09-10T20:32:03.457813+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:03.457813+0000","last_undegraded":"2024-09-10T20:32:03.457813+0000","last_fullsized":"2024-09-10T20:32:03.457813+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16118,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455567+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:03.455567+0000","last_peered":"2024-09-10T20:32:03.455567+0000","last_clean":"2024-09-10T20:32:03.455567+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:03.455567+0000","last_undegraded":"2024-09-10T20:32:03.455567+0000","last_fullsized":"2024-09-10T20:32:03.455567+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12508,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457300+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:03.457300+0000","last_peered":"2024-09-10T20:32:03.457300+0000","last_clean":"2024-09-10T20:32:03.457300+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:03.457300+0000","last_undegraded":"2024-09-10T20:32:03.457300+0000","last_fullsized":"2024-09-10T20:32:03.457300+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457755+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:03.457755+0000","last_peered":"2024-09-10T20:32:03.457755+0000","last_clean":"2024-09-10T20:32:03.457755+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:03.457755+0000","last_undegraded":"2024-09-10T20:32:03.457755+0000","last_fullsized":"2024-09-10T20:32:03.457755+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16151,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455848+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:03.455848+0000","last_peered":"2024-09-10T20:32:03.455848+0000","last_clean":"2024-09-10T20:32:03.455848+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:03.455848+0000","last_undegraded":"2024-09-10T20:32:03.455848+0000","last_fullsized":"2024-09-10T20:32:03.455848+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12421,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.457723+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:32:03.457723+0000","last_peered":"2024-09-10T20:32:03.457723+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:03.457723+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12499,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457681+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:03.457681+0000","last_peered":"2024-09-10T20:32:03.457681+0000","last_clean":"2024-09-10T20:32:03.457681+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:03.457681+0000","last_undegraded":"2024-09-10T20:32:03.457681+0000","last_fullsized":"2024-09-10T20:32:03.457681+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12054,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457177+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:03.457177+0000","last_peered":"2024-09-10T20:32:03.457177+0000","last_clean":"2024-09-10T20:32:03.457177+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:03.457177+0000","last_undegraded":"2024-09-10T20:32:03.457177+0000","last_fullsized":"2024-09-10T20:32:03.457177+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.461644+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:03.461644+0000","last_peered":"2024-09-10T20:32:03.461644+0000","last_clean":"2024-09-10T20:32:03.461644+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:03.461644+0000","last_undegraded":"2024-09-10T20:32:03.461644+0000","last_fullsized":"2024-09-10T20:32:03.461644+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12705,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457390+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:03.457390+0000","last_peered":"2024-09-10T20:32:03.457390+0000","last_clean":"2024-09-10T20:32:03.457390+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:03.457390+0000","last_undegraded":"2024-09-10T20:32:03.457390+0000","last_fullsized":"2024-09-10T20:32:03.457390+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16257,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457564+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:03.457564+0000","last_peered":"2024-09-10T20:32:03.457564+0000","last_clean":"2024-09-10T20:32:03.457564+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:03.457564+0000","last_undegraded":"2024-09-10T20:32:03.457564+0000","last_fullsized":"2024-09-10T20:32:03.457564+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699952,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671888,"statfs":{"total":96636764160,"available":95920013312,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703600,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668240,"statfs":{"total":96636764160,"available":95916277760,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591672,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780168,"statfs":{"total":96636764160,"available":96030892032,"internally_reserved":0,"allocated":499712,"data_stored":198892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704052,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667788,"statfs":{"total":96636764160,"available":95915814912,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:14.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:14.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12442, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:03.458272+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:32:03.458272+0000', 'last_peered': '2024-09-10T20:32:03.458272+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:32:03.458272+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:14.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-10T20:32:14.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "44'12167", 'reported_seq': 15557, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:10.783527+0000', 'last_change': '2024-09-10T20:32:02.343201+0000', 'last_active': '2024-09-10T20:32:10.783527+0000', 'last_peered': '2024-09-10T20:32:10.783527+0000', 'last_clean': '2024-09-10T20:19:05.429266+0000', 'last_became_active': '2024-09-10T20:20:16.137293+0000', 'last_became_peered': '2024-09-10T20:20:16.137293+0000', 'last_unstale': '2024-09-10T20:32:10.783527+0000', 'last_undegraded': '2024-09-10T20:20:15.518784+0000', 'last_fullsized': '2024-09-10T20:20:15.454454+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 342, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 48669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1642, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:14.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:32:14.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 12421, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:03.457723+0000', 'last_change': '2024-09-10T20:20:16.144466+0000', 'last_active': '2024-09-10T20:32:03.457723+0000', 'last_peered': '2024-09-10T20:32:03.457723+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:32:03.457723+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:14.576 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:14.955 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:14.955 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4056/24 objects degraded (16900.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":823,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40416,"bytes_used":2764058624,"bytes_avail":383782998016,"bytes_total":386547056640,"degraded_objects":4056,"degraded_total":24,"degraded_ratio":169},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-09-10T20:32:12.153719+0000","services":{}},"progress_events":{}} 2024-09-10T20:32:14.969 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:32:14.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:15.269 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:15.269 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:15.281 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-10T20:32:14.153977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699276,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374788084,"statfs":{"total":386547056640,"available":383782998016,"internally_reserved":0,"allocated":1961984,"data_stored":758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2761999926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12139,"num_write_kb":-48551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-995,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001337"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12442,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.458272+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:03.458272+0000","last_peered":"2024-09-10T20:32:03.458272+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:03.458272+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12422,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457636+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:03.457636+0000","last_peered":"2024-09-10T20:32:03.457636+0000","last_clean":"2024-09-10T20:32:03.457636+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:03.457636+0000","last_undegraded":"2024-09-10T20:32:03.457636+0000","last_fullsized":"2024-09-10T20:32:03.457636+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16217,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:04.557049+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:04.557049+0000","last_peered":"2024-09-10T20:32:04.557049+0000","last_clean":"2024-09-10T20:32:04.557049+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:04.557049+0000","last_undegraded":"2024-09-10T20:32:04.557049+0000","last_fullsized":"2024-09-10T20:32:04.557049+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16614,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457171+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:03.457171+0000","last_peered":"2024-09-10T20:32:03.457171+0000","last_clean":"2024-09-10T20:32:03.457171+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:03.457171+0000","last_undegraded":"2024-09-10T20:32:03.457171+0000","last_fullsized":"2024-09-10T20:32:03.457171+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":15557,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:10.783527+0000","last_change":"2024-09-10T20:32:02.343201+0000","last_active":"2024-09-10T20:32:10.783527+0000","last_peered":"2024-09-10T20:32:10.783527+0000","last_clean":"2024-09-10T20:19:05.429266+0000","last_became_active":"2024-09-10T20:20:16.137293+0000","last_became_peered":"2024-09-10T20:20:16.137293+0000","last_unstale":"2024-09-10T20:32:10.783527+0000","last_undegraded":"2024-09-10T20:20:15.518784+0000","last_fullsized":"2024-09-10T20:20:15.454454+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1642,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16082,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457813+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:03.457813+0000","last_peered":"2024-09-10T20:32:03.457813+0000","last_clean":"2024-09-10T20:32:03.457813+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:03.457813+0000","last_undegraded":"2024-09-10T20:32:03.457813+0000","last_fullsized":"2024-09-10T20:32:03.457813+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16118,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455567+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:03.455567+0000","last_peered":"2024-09-10T20:32:03.455567+0000","last_clean":"2024-09-10T20:32:03.455567+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:03.455567+0000","last_undegraded":"2024-09-10T20:32:03.455567+0000","last_fullsized":"2024-09-10T20:32:03.455567+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12508,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457300+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:03.457300+0000","last_peered":"2024-09-10T20:32:03.457300+0000","last_clean":"2024-09-10T20:32:03.457300+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:03.457300+0000","last_undegraded":"2024-09-10T20:32:03.457300+0000","last_fullsized":"2024-09-10T20:32:03.457300+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457755+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:03.457755+0000","last_peered":"2024-09-10T20:32:03.457755+0000","last_clean":"2024-09-10T20:32:03.457755+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:03.457755+0000","last_undegraded":"2024-09-10T20:32:03.457755+0000","last_fullsized":"2024-09-10T20:32:03.457755+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16151,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.455848+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:03.455848+0000","last_peered":"2024-09-10T20:32:03.455848+0000","last_clean":"2024-09-10T20:32:03.455848+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:03.455848+0000","last_undegraded":"2024-09-10T20:32:03.455848+0000","last_fullsized":"2024-09-10T20:32:03.455848+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":12421,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:03.457723+0000","last_change":"2024-09-10T20:20:16.144466+0000","last_active":"2024-09-10T20:32:03.457723+0000","last_peered":"2024-09-10T20:32:03.457723+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:03.457723+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12499,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457681+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:03.457681+0000","last_peered":"2024-09-10T20:32:03.457681+0000","last_clean":"2024-09-10T20:32:03.457681+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:03.457681+0000","last_undegraded":"2024-09-10T20:32:03.457681+0000","last_fullsized":"2024-09-10T20:32:03.457681+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12054,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457177+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:03.457177+0000","last_peered":"2024-09-10T20:32:03.457177+0000","last_clean":"2024-09-10T20:32:03.457177+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:03.457177+0000","last_undegraded":"2024-09-10T20:32:03.457177+0000","last_fullsized":"2024-09-10T20:32:03.457177+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.461644+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:03.461644+0000","last_peered":"2024-09-10T20:32:03.461644+0000","last_clean":"2024-09-10T20:32:03.461644+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:03.461644+0000","last_undegraded":"2024-09-10T20:32:03.461644+0000","last_fullsized":"2024-09-10T20:32:03.461644+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12705,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457390+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:03.457390+0000","last_peered":"2024-09-10T20:32:03.457390+0000","last_clean":"2024-09-10T20:32:03.457390+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:03.457390+0000","last_undegraded":"2024-09-10T20:32:03.457390+0000","last_fullsized":"2024-09-10T20:32:03.457390+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16257,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T20:32:03.457564+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:03.457564+0000","last_peered":"2024-09-10T20:32:03.457564+0000","last_clean":"2024-09-10T20:32:03.457564+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:03.457564+0000","last_undegraded":"2024-09-10T20:32:03.457564+0000","last_fullsized":"2024-09-10T20:32:03.457564+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40416,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81224,"num_write_kb":238745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699952,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671888,"statfs":{"total":96636764160,"available":95920013312,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703600,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668240,"statfs":{"total":96636764160,"available":95916277760,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591672,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780168,"statfs":{"total":96636764160,"available":96030892032,"internally_reserved":0,"allocated":499712,"data_stored":198892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704052,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667788,"statfs":{"total":96636764160,"available":95915814912,"internally_reserved":0,"allocated":487424,"data_stored":186604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:18.283 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:18.640 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:18.640 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:18.653 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-10T20:32:18.154557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707660,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779700,"statfs":{"total":386547056640,"available":383774412800,"internally_reserved":0,"allocated":2015232,"data_stored":771468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12165,"num_write_kb":-48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001214"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12446,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:13.492514+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:13.492514+0000","last_peered":"2024-09-10T20:32:13.492514+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:13.492514+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12426,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.495046+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:13.495046+0000","last_peered":"2024-09-10T20:32:13.495046+0000","last_clean":"2024-09-10T20:32:13.495046+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:13.495046+0000","last_undegraded":"2024-09-10T20:32:13.495046+0000","last_fullsized":"2024-09-10T20:32:13.495046+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16618,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491671+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:13.491671+0000","last_peered":"2024-09-10T20:32:13.491671+0000","last_clean":"2024-09-10T20:32:13.491671+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:13.491671+0000","last_undegraded":"2024-09-10T20:32:13.491671+0000","last_fullsized":"2024-09-10T20:32:13.491671+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.667012+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:13.667012+0000","last_peered":"2024-09-10T20:32:13.667012+0000","last_clean":"2024-09-10T20:32:13.667012+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:13.667012+0000","last_undegraded":"2024-09-10T20:32:13.667012+0000","last_fullsized":"2024-09-10T20:32:13.667012+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16086,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492437+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:13.492437+0000","last_peered":"2024-09-10T20:32:13.492437+0000","last_clean":"2024-09-10T20:32:13.492437+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:13.492437+0000","last_undegraded":"2024-09-10T20:32:13.492437+0000","last_fullsized":"2024-09-10T20:32:13.492437+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16122,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.490020+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:13.490020+0000","last_peered":"2024-09-10T20:32:13.490020+0000","last_clean":"2024-09-10T20:32:13.490020+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:13.490020+0000","last_undegraded":"2024-09-10T20:32:13.490020+0000","last_fullsized":"2024-09-10T20:32:13.490020+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12512,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491794+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:13.491794+0000","last_peered":"2024-09-10T20:32:13.491794+0000","last_clean":"2024-09-10T20:32:13.491794+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:13.491794+0000","last_undegraded":"2024-09-10T20:32:13.491794+0000","last_fullsized":"2024-09-10T20:32:13.491794+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16015,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492533+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:13.492533+0000","last_peered":"2024-09-10T20:32:13.492533+0000","last_clean":"2024-09-10T20:32:13.492533+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:13.492533+0000","last_undegraded":"2024-09-10T20:32:13.492533+0000","last_fullsized":"2024-09-10T20:32:13.492533+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":13582,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:15.781312+0000","last_change":"2024-09-10T20:32:12.153008+0000","last_active":"2024-09-10T20:32:15.781312+0000","last_peered":"2024-09-10T20:32:15.781312+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:15.781312+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12503,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491972+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:13.491972+0000","last_peered":"2024-09-10T20:32:13.491972+0000","last_clean":"2024-09-10T20:32:13.491972+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:13.491972+0000","last_undegraded":"2024-09-10T20:32:13.491972+0000","last_fullsized":"2024-09-10T20:32:13.491972+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12058,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491678+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:13.491678+0000","last_peered":"2024-09-10T20:32:13.491678+0000","last_clean":"2024-09-10T20:32:13.491678+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:13.491678+0000","last_undegraded":"2024-09-10T20:32:13.491678+0000","last_fullsized":"2024-09-10T20:32:13.491678+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16185,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491787+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:13.491787+0000","last_peered":"2024-09-10T20:32:13.491787+0000","last_clean":"2024-09-10T20:32:13.491787+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:13.491787+0000","last_undegraded":"2024-09-10T20:32:13.491787+0000","last_fullsized":"2024-09-10T20:32:13.491787+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12709,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491797+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:13.491797+0000","last_peered":"2024-09-10T20:32:13.491797+0000","last_clean":"2024-09-10T20:32:13.491797+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:13.491797+0000","last_undegraded":"2024-09-10T20:32:13.491797+0000","last_fullsized":"2024-09-10T20:32:13.491797+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16261,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491897+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:13.491897+0000","last_peered":"2024-09-10T20:32:13.491897+0000","last_clean":"2024-09-10T20:32:13.491897+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:13.491897+0000","last_undegraded":"2024-09-10T20:32:13.491897+0000","last_fullsized":"2024-09-10T20:32:13.491897+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699972,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671868,"statfs":{"total":96636764160,"available":95919992832,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600004,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771836,"statfs":{"total":96636764160,"available":96022360064,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:18.654 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:18.962 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:18.962 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:18.974 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-10T20:32:18.154557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707660,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779700,"statfs":{"total":386547056640,"available":383774412800,"internally_reserved":0,"allocated":2015232,"data_stored":771468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12165,"num_write_kb":-48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001214"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12446,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:13.492514+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:13.492514+0000","last_peered":"2024-09-10T20:32:13.492514+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:13.492514+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12426,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.495046+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:13.495046+0000","last_peered":"2024-09-10T20:32:13.495046+0000","last_clean":"2024-09-10T20:32:13.495046+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:13.495046+0000","last_undegraded":"2024-09-10T20:32:13.495046+0000","last_fullsized":"2024-09-10T20:32:13.495046+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16618,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491671+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:13.491671+0000","last_peered":"2024-09-10T20:32:13.491671+0000","last_clean":"2024-09-10T20:32:13.491671+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:13.491671+0000","last_undegraded":"2024-09-10T20:32:13.491671+0000","last_fullsized":"2024-09-10T20:32:13.491671+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.667012+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:13.667012+0000","last_peered":"2024-09-10T20:32:13.667012+0000","last_clean":"2024-09-10T20:32:13.667012+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:13.667012+0000","last_undegraded":"2024-09-10T20:32:13.667012+0000","last_fullsized":"2024-09-10T20:32:13.667012+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16086,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492437+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:13.492437+0000","last_peered":"2024-09-10T20:32:13.492437+0000","last_clean":"2024-09-10T20:32:13.492437+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:13.492437+0000","last_undegraded":"2024-09-10T20:32:13.492437+0000","last_fullsized":"2024-09-10T20:32:13.492437+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16122,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.490020+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:13.490020+0000","last_peered":"2024-09-10T20:32:13.490020+0000","last_clean":"2024-09-10T20:32:13.490020+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:13.490020+0000","last_undegraded":"2024-09-10T20:32:13.490020+0000","last_fullsized":"2024-09-10T20:32:13.490020+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12512,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491794+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:13.491794+0000","last_peered":"2024-09-10T20:32:13.491794+0000","last_clean":"2024-09-10T20:32:13.491794+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:13.491794+0000","last_undegraded":"2024-09-10T20:32:13.491794+0000","last_fullsized":"2024-09-10T20:32:13.491794+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16015,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492533+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:13.492533+0000","last_peered":"2024-09-10T20:32:13.492533+0000","last_clean":"2024-09-10T20:32:13.492533+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:13.492533+0000","last_undegraded":"2024-09-10T20:32:13.492533+0000","last_fullsized":"2024-09-10T20:32:13.492533+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":13582,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:15.781312+0000","last_change":"2024-09-10T20:32:12.153008+0000","last_active":"2024-09-10T20:32:15.781312+0000","last_peered":"2024-09-10T20:32:15.781312+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:15.781312+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12503,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491972+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:13.491972+0000","last_peered":"2024-09-10T20:32:13.491972+0000","last_clean":"2024-09-10T20:32:13.491972+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:13.491972+0000","last_undegraded":"2024-09-10T20:32:13.491972+0000","last_fullsized":"2024-09-10T20:32:13.491972+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12058,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491678+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:13.491678+0000","last_peered":"2024-09-10T20:32:13.491678+0000","last_clean":"2024-09-10T20:32:13.491678+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:13.491678+0000","last_undegraded":"2024-09-10T20:32:13.491678+0000","last_fullsized":"2024-09-10T20:32:13.491678+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16185,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491787+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:13.491787+0000","last_peered":"2024-09-10T20:32:13.491787+0000","last_clean":"2024-09-10T20:32:13.491787+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:13.491787+0000","last_undegraded":"2024-09-10T20:32:13.491787+0000","last_fullsized":"2024-09-10T20:32:13.491787+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12709,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491797+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:13.491797+0000","last_peered":"2024-09-10T20:32:13.491797+0000","last_clean":"2024-09-10T20:32:13.491797+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:13.491797+0000","last_undegraded":"2024-09-10T20:32:13.491797+0000","last_fullsized":"2024-09-10T20:32:13.491797+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16261,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491897+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:13.491897+0000","last_peered":"2024-09-10T20:32:13.491897+0000","last_clean":"2024-09-10T20:32:13.491897+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:13.491897+0000","last_undegraded":"2024-09-10T20:32:13.491897+0000","last_fullsized":"2024-09-10T20:32:13.491897+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699972,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671868,"statfs":{"total":96636764160,"available":95919992832,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600004,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771836,"statfs":{"total":96636764160,"available":96022360064,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:18.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:18.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12446, 'reported_epoch': 60, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:13.492514+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:32:13.492514+0000', 'last_peered': '2024-09-10T20:32:13.492514+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:32:13.492514+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:18.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:32:18.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 13582, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:15.781312+0000', 'last_change': '2024-09-10T20:32:12.153008+0000', 'last_active': '2024-09-10T20:32:15.781312+0000', 'last_peered': '2024-09-10T20:32:15.781312+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:32:15.781312+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 681, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:18.976 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:19.342 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:19.343 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3136/16 objects degraded (19600.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":828,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":26944,"bytes_used":2772643840,"bytes_avail":383774412800,"bytes_total":386547056640,"degraded_objects":3136,"degraded_total":16,"degraded_ratio":196},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-09-10T20:32:12.153719+0000","services":{}},"progress_events":{}} 2024-09-10T20:32:19.356 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:32:19.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:19.678 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:19.692 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-10T20:32:18.154557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707660,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779700,"statfs":{"total":386547056640,"available":383774412800,"internally_reserved":0,"allocated":2015232,"data_stored":771468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12165,"num_write_kb":-48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001214"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12446,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:13.492514+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:13.492514+0000","last_peered":"2024-09-10T20:32:13.492514+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:13.492514+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12426,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.495046+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:13.495046+0000","last_peered":"2024-09-10T20:32:13.495046+0000","last_clean":"2024-09-10T20:32:13.495046+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:13.495046+0000","last_undegraded":"2024-09-10T20:32:13.495046+0000","last_fullsized":"2024-09-10T20:32:13.495046+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16618,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491671+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:13.491671+0000","last_peered":"2024-09-10T20:32:13.491671+0000","last_clean":"2024-09-10T20:32:13.491671+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:13.491671+0000","last_undegraded":"2024-09-10T20:32:13.491671+0000","last_fullsized":"2024-09-10T20:32:13.491671+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.667012+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:13.667012+0000","last_peered":"2024-09-10T20:32:13.667012+0000","last_clean":"2024-09-10T20:32:13.667012+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:13.667012+0000","last_undegraded":"2024-09-10T20:32:13.667012+0000","last_fullsized":"2024-09-10T20:32:13.667012+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16086,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492437+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:13.492437+0000","last_peered":"2024-09-10T20:32:13.492437+0000","last_clean":"2024-09-10T20:32:13.492437+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:13.492437+0000","last_undegraded":"2024-09-10T20:32:13.492437+0000","last_fullsized":"2024-09-10T20:32:13.492437+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16122,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.490020+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:13.490020+0000","last_peered":"2024-09-10T20:32:13.490020+0000","last_clean":"2024-09-10T20:32:13.490020+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:13.490020+0000","last_undegraded":"2024-09-10T20:32:13.490020+0000","last_fullsized":"2024-09-10T20:32:13.490020+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12512,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491794+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:13.491794+0000","last_peered":"2024-09-10T20:32:13.491794+0000","last_clean":"2024-09-10T20:32:13.491794+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:13.491794+0000","last_undegraded":"2024-09-10T20:32:13.491794+0000","last_fullsized":"2024-09-10T20:32:13.491794+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16015,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492533+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:13.492533+0000","last_peered":"2024-09-10T20:32:13.492533+0000","last_clean":"2024-09-10T20:32:13.492533+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:13.492533+0000","last_undegraded":"2024-09-10T20:32:13.492533+0000","last_fullsized":"2024-09-10T20:32:13.492533+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":13582,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:15.781312+0000","last_change":"2024-09-10T20:32:12.153008+0000","last_active":"2024-09-10T20:32:15.781312+0000","last_peered":"2024-09-10T20:32:15.781312+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:15.781312+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12503,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491972+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:13.491972+0000","last_peered":"2024-09-10T20:32:13.491972+0000","last_clean":"2024-09-10T20:32:13.491972+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:13.491972+0000","last_undegraded":"2024-09-10T20:32:13.491972+0000","last_fullsized":"2024-09-10T20:32:13.491972+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12058,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491678+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:13.491678+0000","last_peered":"2024-09-10T20:32:13.491678+0000","last_clean":"2024-09-10T20:32:13.491678+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:13.491678+0000","last_undegraded":"2024-09-10T20:32:13.491678+0000","last_fullsized":"2024-09-10T20:32:13.491678+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16185,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491787+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:13.491787+0000","last_peered":"2024-09-10T20:32:13.491787+0000","last_clean":"2024-09-10T20:32:13.491787+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:13.491787+0000","last_undegraded":"2024-09-10T20:32:13.491787+0000","last_fullsized":"2024-09-10T20:32:13.491787+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12709,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491797+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:13.491797+0000","last_peered":"2024-09-10T20:32:13.491797+0000","last_clean":"2024-09-10T20:32:13.491797+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:13.491797+0000","last_undegraded":"2024-09-10T20:32:13.491797+0000","last_fullsized":"2024-09-10T20:32:13.491797+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16261,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491897+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:13.491897+0000","last_peered":"2024-09-10T20:32:13.491897+0000","last_clean":"2024-09-10T20:32:13.491897+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:13.491897+0000","last_undegraded":"2024-09-10T20:32:13.491897+0000","last_fullsized":"2024-09-10T20:32:13.491897+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699972,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671868,"statfs":{"total":96636764160,"available":95919992832,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600004,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771836,"statfs":{"total":96636764160,"available":96022360064,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:22.694 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:23.013 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:23.014 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:23.026 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-10T20:32:22.522953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699332,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374788028,"statfs":{"total":386547056640,"available":383782940672,"internally_reserved":0,"allocated":2015232,"data_stored":771468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2761999926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12165,"num_write_kb":-48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.034293"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12446,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:13.492514+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:13.492514+0000","last_peered":"2024-09-10T20:32:13.492514+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:13.492514+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12426,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.495046+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:13.495046+0000","last_peered":"2024-09-10T20:32:13.495046+0000","last_clean":"2024-09-10T20:32:13.495046+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:13.495046+0000","last_undegraded":"2024-09-10T20:32:13.495046+0000","last_fullsized":"2024-09-10T20:32:13.495046+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16618,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491671+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:13.491671+0000","last_peered":"2024-09-10T20:32:13.491671+0000","last_clean":"2024-09-10T20:32:13.491671+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:13.491671+0000","last_undegraded":"2024-09-10T20:32:13.491671+0000","last_fullsized":"2024-09-10T20:32:13.491671+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.667012+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:13.667012+0000","last_peered":"2024-09-10T20:32:13.667012+0000","last_clean":"2024-09-10T20:32:13.667012+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:13.667012+0000","last_undegraded":"2024-09-10T20:32:13.667012+0000","last_fullsized":"2024-09-10T20:32:13.667012+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16086,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492437+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:13.492437+0000","last_peered":"2024-09-10T20:32:13.492437+0000","last_clean":"2024-09-10T20:32:13.492437+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:13.492437+0000","last_undegraded":"2024-09-10T20:32:13.492437+0000","last_fullsized":"2024-09-10T20:32:13.492437+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16122,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.490020+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:13.490020+0000","last_peered":"2024-09-10T20:32:13.490020+0000","last_clean":"2024-09-10T20:32:13.490020+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:13.490020+0000","last_undegraded":"2024-09-10T20:32:13.490020+0000","last_fullsized":"2024-09-10T20:32:13.490020+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12512,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491794+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:13.491794+0000","last_peered":"2024-09-10T20:32:13.491794+0000","last_clean":"2024-09-10T20:32:13.491794+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:13.491794+0000","last_undegraded":"2024-09-10T20:32:13.491794+0000","last_fullsized":"2024-09-10T20:32:13.491794+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16015,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492533+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:13.492533+0000","last_peered":"2024-09-10T20:32:13.492533+0000","last_clean":"2024-09-10T20:32:13.492533+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:13.492533+0000","last_undegraded":"2024-09-10T20:32:13.492533+0000","last_fullsized":"2024-09-10T20:32:13.492533+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":15607,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:20.783221+0000","last_change":"2024-09-10T20:32:12.153008+0000","last_active":"2024-09-10T20:32:20.783221+0000","last_peered":"2024-09-10T20:32:20.783221+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:20.783221+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12503,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491972+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:13.491972+0000","last_peered":"2024-09-10T20:32:13.491972+0000","last_clean":"2024-09-10T20:32:13.491972+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:13.491972+0000","last_undegraded":"2024-09-10T20:32:13.491972+0000","last_fullsized":"2024-09-10T20:32:13.491972+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12058,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491678+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:13.491678+0000","last_peered":"2024-09-10T20:32:13.491678+0000","last_clean":"2024-09-10T20:32:13.491678+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:13.491678+0000","last_undegraded":"2024-09-10T20:32:13.491678+0000","last_fullsized":"2024-09-10T20:32:13.491678+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16185,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491787+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:13.491787+0000","last_peered":"2024-09-10T20:32:13.491787+0000","last_clean":"2024-09-10T20:32:13.491787+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:13.491787+0000","last_undegraded":"2024-09-10T20:32:13.491787+0000","last_fullsized":"2024-09-10T20:32:13.491787+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12709,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491797+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:13.491797+0000","last_peered":"2024-09-10T20:32:13.491797+0000","last_clean":"2024-09-10T20:32:13.491797+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:13.491797+0000","last_undegraded":"2024-09-10T20:32:13.491797+0000","last_fullsized":"2024-09-10T20:32:13.491797+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16261,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491897+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:13.491897+0000","last_peered":"2024-09-10T20:32:13.491897+0000","last_clean":"2024-09-10T20:32:13.491897+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:13.491897+0000","last_undegraded":"2024-09-10T20:32:13.491897+0000","last_fullsized":"2024-09-10T20:32:13.491897+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699972,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671868,"statfs":{"total":96636764160,"available":95919992832,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591676,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780164,"statfs":{"total":96636764160,"available":96030887936,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:23.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:23.414 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:23.425 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-10T20:32:22.522953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699332,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374788028,"statfs":{"total":386547056640,"available":383782940672,"internally_reserved":0,"allocated":2015232,"data_stored":771468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2761999926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12165,"num_write_kb":-48669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.034293"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12446,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:13.492514+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:13.492514+0000","last_peered":"2024-09-10T20:32:13.492514+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:13.492514+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12426,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.495046+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:13.495046+0000","last_peered":"2024-09-10T20:32:13.495046+0000","last_clean":"2024-09-10T20:32:13.495046+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:13.495046+0000","last_undegraded":"2024-09-10T20:32:13.495046+0000","last_fullsized":"2024-09-10T20:32:13.495046+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16618,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491671+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:13.491671+0000","last_peered":"2024-09-10T20:32:13.491671+0000","last_clean":"2024-09-10T20:32:13.491671+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:13.491671+0000","last_undegraded":"2024-09-10T20:32:13.491671+0000","last_fullsized":"2024-09-10T20:32:13.491671+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.667012+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:13.667012+0000","last_peered":"2024-09-10T20:32:13.667012+0000","last_clean":"2024-09-10T20:32:13.667012+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:13.667012+0000","last_undegraded":"2024-09-10T20:32:13.667012+0000","last_fullsized":"2024-09-10T20:32:13.667012+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16086,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492437+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:13.492437+0000","last_peered":"2024-09-10T20:32:13.492437+0000","last_clean":"2024-09-10T20:32:13.492437+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:13.492437+0000","last_undegraded":"2024-09-10T20:32:13.492437+0000","last_fullsized":"2024-09-10T20:32:13.492437+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16122,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.490020+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:13.490020+0000","last_peered":"2024-09-10T20:32:13.490020+0000","last_clean":"2024-09-10T20:32:13.490020+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:13.490020+0000","last_undegraded":"2024-09-10T20:32:13.490020+0000","last_fullsized":"2024-09-10T20:32:13.490020+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12512,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491794+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:13.491794+0000","last_peered":"2024-09-10T20:32:13.491794+0000","last_clean":"2024-09-10T20:32:13.491794+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:13.491794+0000","last_undegraded":"2024-09-10T20:32:13.491794+0000","last_fullsized":"2024-09-10T20:32:13.491794+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16015,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492533+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:13.492533+0000","last_peered":"2024-09-10T20:32:13.492533+0000","last_clean":"2024-09-10T20:32:13.492533+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:13.492533+0000","last_undegraded":"2024-09-10T20:32:13.492533+0000","last_fullsized":"2024-09-10T20:32:13.492533+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":15607,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:20.783221+0000","last_change":"2024-09-10T20:32:12.153008+0000","last_active":"2024-09-10T20:32:20.783221+0000","last_peered":"2024-09-10T20:32:20.783221+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:20.783221+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12503,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491972+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:13.491972+0000","last_peered":"2024-09-10T20:32:13.491972+0000","last_clean":"2024-09-10T20:32:13.491972+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:13.491972+0000","last_undegraded":"2024-09-10T20:32:13.491972+0000","last_fullsized":"2024-09-10T20:32:13.491972+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12058,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491678+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:13.491678+0000","last_peered":"2024-09-10T20:32:13.491678+0000","last_clean":"2024-09-10T20:32:13.491678+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:13.491678+0000","last_undegraded":"2024-09-10T20:32:13.491678+0000","last_fullsized":"2024-09-10T20:32:13.491678+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16185,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491787+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:13.491787+0000","last_peered":"2024-09-10T20:32:13.491787+0000","last_clean":"2024-09-10T20:32:13.491787+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:13.491787+0000","last_undegraded":"2024-09-10T20:32:13.491787+0000","last_fullsized":"2024-09-10T20:32:13.491787+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12709,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491797+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:13.491797+0000","last_peered":"2024-09-10T20:32:13.491797+0000","last_clean":"2024-09-10T20:32:13.491797+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:13.491797+0000","last_undegraded":"2024-09-10T20:32:13.491797+0000","last_fullsized":"2024-09-10T20:32:13.491797+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16261,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491897+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:13.491897+0000","last_peered":"2024-09-10T20:32:13.491897+0000","last_clean":"2024-09-10T20:32:13.491897+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:13.491897+0000","last_undegraded":"2024-09-10T20:32:13.491897+0000","last_fullsized":"2024-09-10T20:32:13.491897+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699972,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671868,"statfs":{"total":96636764160,"available":95919992832,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591676,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780164,"statfs":{"total":96636764160,"available":96030887936,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:23.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:23.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 12446, 'reported_epoch': 60, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:13.492514+0000', 'last_change': '2024-09-10T20:20:16.160830+0000', 'last_active': '2024-09-10T20:32:13.492514+0000', 'last_peered': '2024-09-10T20:32:13.492514+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:32:13.492514+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:23.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-10T20:32:23.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "44'12110", 'reported_seq': 15607, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:20.783221+0000', 'last_change': '2024-09-10T20:32:12.153008+0000', 'last_active': '2024-09-10T20:32:20.783221+0000', 'last_peered': '2024-09-10T20:32:20.783221+0000', 'last_clean': '2024-09-10T20:19:06.262108+0000', 'last_became_active': '2024-09-10T20:20:16.137808+0000', 'last_became_peered': '2024-09-10T20:20:16.137808+0000', 'last_unstale': '2024-09-10T20:32:20.783221+0000', 'last_undegraded': '2024-09-10T20:20:15.522301+0000', 'last_fullsized': '2024-09-10T20:20:15.457111+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 48436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1693, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -266}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:23.427 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:23.876 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:23.877 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2124/16 objects degraded (13275.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":832,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":62,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":26944,"bytes_used":2764115968,"bytes_avail":383782940672,"bytes_total":386547056640,"degraded_objects":2124,"degraded_total":16,"degraded_ratio":132.75},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-09-10T20:32:12.153719+0000","services":{}},"progress_events":{}} 2024-09-10T20:32:23.890 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:32:23.890 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:24.199 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:24.200 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:24.213 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-10T20:32:24.155466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2699332,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374788028,"statfs":{"total":386547056640,"available":383782940672,"internally_reserved":0,"allocated":2015232,"data_stored":771468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2761999926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001215"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":12446,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:13.492514+0000","last_change":"2024-09-10T20:20:16.160830+0000","last_active":"2024-09-10T20:32:13.492514+0000","last_peered":"2024-09-10T20:32:13.492514+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:13.492514+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12426,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.495046+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:13.495046+0000","last_peered":"2024-09-10T20:32:13.495046+0000","last_clean":"2024-09-10T20:32:13.495046+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:13.495046+0000","last_undegraded":"2024-09-10T20:32:13.495046+0000","last_fullsized":"2024-09-10T20:32:13.495046+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16618,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491671+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:13.491671+0000","last_peered":"2024-09-10T20:32:13.491671+0000","last_clean":"2024-09-10T20:32:13.491671+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:13.491671+0000","last_undegraded":"2024-09-10T20:32:13.491671+0000","last_fullsized":"2024-09-10T20:32:13.491671+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.667012+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:13.667012+0000","last_peered":"2024-09-10T20:32:13.667012+0000","last_clean":"2024-09-10T20:32:13.667012+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:13.667012+0000","last_undegraded":"2024-09-10T20:32:13.667012+0000","last_fullsized":"2024-09-10T20:32:13.667012+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16086,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492437+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:13.492437+0000","last_peered":"2024-09-10T20:32:13.492437+0000","last_clean":"2024-09-10T20:32:13.492437+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:13.492437+0000","last_undegraded":"2024-09-10T20:32:13.492437+0000","last_fullsized":"2024-09-10T20:32:13.492437+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16122,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.490020+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:13.490020+0000","last_peered":"2024-09-10T20:32:13.490020+0000","last_clean":"2024-09-10T20:32:13.490020+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:13.490020+0000","last_undegraded":"2024-09-10T20:32:13.490020+0000","last_fullsized":"2024-09-10T20:32:13.490020+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12512,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491794+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:13.491794+0000","last_peered":"2024-09-10T20:32:13.491794+0000","last_clean":"2024-09-10T20:32:13.491794+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:13.491794+0000","last_undegraded":"2024-09-10T20:32:13.491794+0000","last_fullsized":"2024-09-10T20:32:13.491794+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16015,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.492533+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:13.492533+0000","last_peered":"2024-09-10T20:32:13.492533+0000","last_clean":"2024-09-10T20:32:13.492533+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:13.492533+0000","last_undegraded":"2024-09-10T20:32:13.492533+0000","last_fullsized":"2024-09-10T20:32:13.492533+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":15607,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:20.783221+0000","last_change":"2024-09-10T20:32:12.153008+0000","last_active":"2024-09-10T20:32:20.783221+0000","last_peered":"2024-09-10T20:32:20.783221+0000","last_clean":"2024-09-10T20:19:06.262108+0000","last_became_active":"2024-09-10T20:20:16.137808+0000","last_became_peered":"2024-09-10T20:20:16.137808+0000","last_unstale":"2024-09-10T20:32:20.783221+0000","last_undegraded":"2024-09-10T20:20:15.522301+0000","last_fullsized":"2024-09-10T20:20:15.457111+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12503,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491972+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:13.491972+0000","last_peered":"2024-09-10T20:32:13.491972+0000","last_clean":"2024-09-10T20:32:13.491972+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:13.491972+0000","last_undegraded":"2024-09-10T20:32:13.491972+0000","last_fullsized":"2024-09-10T20:32:13.491972+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12058,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491678+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:13.491678+0000","last_peered":"2024-09-10T20:32:13.491678+0000","last_clean":"2024-09-10T20:32:13.491678+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:13.491678+0000","last_undegraded":"2024-09-10T20:32:13.491678+0000","last_fullsized":"2024-09-10T20:32:13.491678+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16185,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491787+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:13.491787+0000","last_peered":"2024-09-10T20:32:13.491787+0000","last_clean":"2024-09-10T20:32:13.491787+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:13.491787+0000","last_undegraded":"2024-09-10T20:32:13.491787+0000","last_fullsized":"2024-09-10T20:32:13.491787+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12709,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491797+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:13.491797+0000","last_peered":"2024-09-10T20:32:13.491797+0000","last_clean":"2024-09-10T20:32:13.491797+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:13.491797+0000","last_undegraded":"2024-09-10T20:32:13.491797+0000","last_fullsized":"2024-09-10T20:32:13.491797+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16261,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491897+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:13.491897+0000","last_peered":"2024-09-10T20:32:13.491897+0000","last_clean":"2024-09-10T20:32:13.491897+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:13.491897+0000","last_undegraded":"2024-09-10T20:32:13.491897+0000","last_fullsized":"2024-09-10T20:32:13.491897+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69059,"num_write_kb":190076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699972,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671868,"statfs":{"total":96636764160,"available":95919992832,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591676,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780164,"statfs":{"total":96636764160,"available":96030887936,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:27.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:27.530 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:27.531 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:27.545 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-10T20:32:26.155773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707696,"kb_used_data":2016,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779664,"statfs":{"total":386547056640,"available":383774375936,"internally_reserved":0,"allocated":2064384,"data_stored":790035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12107,"num_write_kb":-48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001216"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":13447,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:25.780757+0000","last_change":"2024-09-10T20:32:21.781527+0000","last_active":"2024-09-10T20:32:25.780757+0000","last_peered":"2024-09-10T20:32:25.780757+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:25.780757+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.536405+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:23.536405+0000","last_peered":"2024-09-10T20:32:23.536405+0000","last_clean":"2024-09-10T20:32:23.536405+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:23.536405+0000","last_undegraded":"2024-09-10T20:32:23.536405+0000","last_fullsized":"2024-09-10T20:32:23.536405+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16622,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534949+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:23.534949+0000","last_peered":"2024-09-10T20:32:23.534949+0000","last_clean":"2024-09-10T20:32:23.534949+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:23.534949+0000","last_undegraded":"2024-09-10T20:32:23.534949+0000","last_fullsized":"2024-09-10T20:32:23.534949+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16266,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534736+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:23.534736+0000","last_peered":"2024-09-10T20:32:23.534736+0000","last_clean":"2024-09-10T20:32:23.534736+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:23.534736+0000","last_undegraded":"2024-09-10T20:32:23.534736+0000","last_fullsized":"2024-09-10T20:32:23.534736+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535142+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:23.535142+0000","last_peered":"2024-09-10T20:32:23.535142+0000","last_clean":"2024-09-10T20:32:23.535142+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:23.535142+0000","last_undegraded":"2024-09-10T20:32:23.535142+0000","last_fullsized":"2024-09-10T20:32:23.535142+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535223+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:23.535223+0000","last_peered":"2024-09-10T20:32:23.535223+0000","last_clean":"2024-09-10T20:32:23.535223+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:23.535223+0000","last_undegraded":"2024-09-10T20:32:23.535223+0000","last_fullsized":"2024-09-10T20:32:23.535223+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:23.535061+0000","last_peered":"2024-09-10T20:32:23.535061+0000","last_clean":"2024-09-10T20:32:23.535061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:23.535061+0000","last_undegraded":"2024-09-10T20:32:23.535061+0000","last_fullsized":"2024-09-10T20:32:23.535061+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16166,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:24.954031+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:24.954031+0000","last_peered":"2024-09-10T20:32:24.954031+0000","last_clean":"2024-09-10T20:32:24.954031+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:24.954031+0000","last_undegraded":"2024-09-10T20:32:24.954031+0000","last_fullsized":"2024-09-10T20:32:24.954031+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12507,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535394+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:23.535394+0000","last_peered":"2024-09-10T20:32:23.535394+0000","last_clean":"2024-09-10T20:32:23.535394+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:23.535394+0000","last_undegraded":"2024-09-10T20:32:23.535394+0000","last_fullsized":"2024-09-10T20:32:23.535394+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12062,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535054+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:23.535054+0000","last_peered":"2024-09-10T20:32:23.535054+0000","last_clean":"2024-09-10T20:32:23.535054+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:23.535054+0000","last_undegraded":"2024-09-10T20:32:23.535054+0000","last_fullsized":"2024-09-10T20:32:23.535054+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535032+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:23.535032+0000","last_peered":"2024-09-10T20:32:23.535032+0000","last_clean":"2024-09-10T20:32:23.535032+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:23.535032+0000","last_undegraded":"2024-09-10T20:32:23.535032+0000","last_fullsized":"2024-09-10T20:32:23.535032+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12713,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535159+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:23.535159+0000","last_peered":"2024-09-10T20:32:23.535159+0000","last_clean":"2024-09-10T20:32:23.535159+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:23.535159+0000","last_undegraded":"2024-09-10T20:32:23.535159+0000","last_fullsized":"2024-09-10T20:32:23.535159+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16265,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535257+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:23.535257+0000","last_peered":"2024-09-10T20:32:23.535257+0000","last_clean":"2024-09-10T20:32:23.535257+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:23.535257+0000","last_undegraded":"2024-09-10T20:32:23.535257+0000","last_fullsized":"2024-09-10T20:32:23.535257+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699980,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671860,"statfs":{"total":96636764160,"available":95919984640,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600012,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771828,"statfs":{"total":96636764160,"available":96022351872,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:27.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:27.847 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:27.848 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:27.860 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-10T20:32:26.155773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707696,"kb_used_data":2016,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779664,"statfs":{"total":386547056640,"available":383774375936,"internally_reserved":0,"allocated":2064384,"data_stored":790035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12107,"num_write_kb":-48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001216"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":13447,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:25.780757+0000","last_change":"2024-09-10T20:32:21.781527+0000","last_active":"2024-09-10T20:32:25.780757+0000","last_peered":"2024-09-10T20:32:25.780757+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:25.780757+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.536405+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:23.536405+0000","last_peered":"2024-09-10T20:32:23.536405+0000","last_clean":"2024-09-10T20:32:23.536405+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:23.536405+0000","last_undegraded":"2024-09-10T20:32:23.536405+0000","last_fullsized":"2024-09-10T20:32:23.536405+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16221,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491078+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:13.491078+0000","last_peered":"2024-09-10T20:32:13.491078+0000","last_clean":"2024-09-10T20:32:13.491078+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:13.491078+0000","last_undegraded":"2024-09-10T20:32:13.491078+0000","last_fullsized":"2024-09-10T20:32:13.491078+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16622,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534949+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:23.534949+0000","last_peered":"2024-09-10T20:32:23.534949+0000","last_clean":"2024-09-10T20:32:23.534949+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:23.534949+0000","last_undegraded":"2024-09-10T20:32:23.534949+0000","last_fullsized":"2024-09-10T20:32:23.534949+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16266,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534736+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:23.534736+0000","last_peered":"2024-09-10T20:32:23.534736+0000","last_clean":"2024-09-10T20:32:23.534736+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:23.534736+0000","last_undegraded":"2024-09-10T20:32:23.534736+0000","last_fullsized":"2024-09-10T20:32:23.534736+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535142+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:23.535142+0000","last_peered":"2024-09-10T20:32:23.535142+0000","last_clean":"2024-09-10T20:32:23.535142+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:23.535142+0000","last_undegraded":"2024-09-10T20:32:23.535142+0000","last_fullsized":"2024-09-10T20:32:23.535142+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535223+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:23.535223+0000","last_peered":"2024-09-10T20:32:23.535223+0000","last_clean":"2024-09-10T20:32:23.535223+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:23.535223+0000","last_undegraded":"2024-09-10T20:32:23.535223+0000","last_fullsized":"2024-09-10T20:32:23.535223+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:23.535061+0000","last_peered":"2024-09-10T20:32:23.535061+0000","last_clean":"2024-09-10T20:32:23.535061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:23.535061+0000","last_undegraded":"2024-09-10T20:32:23.535061+0000","last_fullsized":"2024-09-10T20:32:23.535061+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16155,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T20:32:13.491191+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:13.491191+0000","last_peered":"2024-09-10T20:32:13.491191+0000","last_clean":"2024-09-10T20:32:13.491191+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:13.491191+0000","last_undegraded":"2024-09-10T20:32:13.491191+0000","last_fullsized":"2024-09-10T20:32:13.491191+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16166,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:24.954031+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:24.954031+0000","last_peered":"2024-09-10T20:32:24.954031+0000","last_clean":"2024-09-10T20:32:24.954031+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:24.954031+0000","last_undegraded":"2024-09-10T20:32:24.954031+0000","last_fullsized":"2024-09-10T20:32:24.954031+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12507,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535394+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:23.535394+0000","last_peered":"2024-09-10T20:32:23.535394+0000","last_clean":"2024-09-10T20:32:23.535394+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:23.535394+0000","last_undegraded":"2024-09-10T20:32:23.535394+0000","last_fullsized":"2024-09-10T20:32:23.535394+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12062,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535054+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:23.535054+0000","last_peered":"2024-09-10T20:32:23.535054+0000","last_clean":"2024-09-10T20:32:23.535054+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:23.535054+0000","last_undegraded":"2024-09-10T20:32:23.535054+0000","last_fullsized":"2024-09-10T20:32:23.535054+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535032+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:23.535032+0000","last_peered":"2024-09-10T20:32:23.535032+0000","last_clean":"2024-09-10T20:32:23.535032+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:23.535032+0000","last_undegraded":"2024-09-10T20:32:23.535032+0000","last_fullsized":"2024-09-10T20:32:23.535032+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12713,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535159+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:23.535159+0000","last_peered":"2024-09-10T20:32:23.535159+0000","last_clean":"2024-09-10T20:32:23.535159+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:23.535159+0000","last_undegraded":"2024-09-10T20:32:23.535159+0000","last_fullsized":"2024-09-10T20:32:23.535159+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16265,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535257+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:23.535257+0000","last_peered":"2024-09-10T20:32:23.535257+0000","last_clean":"2024-09-10T20:32:23.535257+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:23.535257+0000","last_undegraded":"2024-09-10T20:32:23.535257+0000","last_fullsized":"2024-09-10T20:32:23.535257+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699980,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671860,"statfs":{"total":96636764160,"available":95919984640,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703620,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668220,"statfs":{"total":96636764160,"available":95916257280,"internally_reserved":0,"allocated":503808,"data_stored":192867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600012,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771828,"statfs":{"total":96636764160,"available":96022351872,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:27.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:27.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 13447, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:25.780757+0000', 'last_change': '2024-09-10T20:32:21.781527+0000', 'last_active': '2024-09-10T20:32:25.780757+0000', 'last_peered': '2024-09-10T20:32:25.780757+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:32:25.780757+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1357, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 609, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1357}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:27.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:28.222 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:28.222 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1357/4 objects degraded (33925.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":836,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":62,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1,"data_bytes":6736,"bytes_used":2772680704,"bytes_avail":383774375936,"bytes_total":386547056640,"degraded_objects":1357,"degraded_total":4,"degraded_ratio":339.25},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-09-10T20:32:12.153719+0000","services":{}},"progress_events":{}} 2024-09-10T20:32:28.237 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:32:28.237 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:28.553 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:28.553 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:28.566 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-10T20:32:28.156090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707712,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2705585,"kb_avail":374779648,"statfs":{"total":386547056640,"available":383774359552,"internally_reserved":0,"allocated":2080768,"data_stored":796224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770519606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12107,"num_write_kb":-48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001254"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":13447,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:25.780757+0000","last_change":"2024-09-10T20:32:21.781527+0000","last_active":"2024-09-10T20:32:25.780757+0000","last_peered":"2024-09-10T20:32:25.780757+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:25.780757+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.536405+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:23.536405+0000","last_peered":"2024-09-10T20:32:23.536405+0000","last_clean":"2024-09-10T20:32:23.536405+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:23.536405+0000","last_undegraded":"2024-09-10T20:32:23.536405+0000","last_fullsized":"2024-09-10T20:32:23.536405+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16225,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534127+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:23.534127+0000","last_peered":"2024-09-10T20:32:23.534127+0000","last_clean":"2024-09-10T20:32:23.534127+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:23.534127+0000","last_undegraded":"2024-09-10T20:32:23.534127+0000","last_fullsized":"2024-09-10T20:32:23.534127+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16622,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534949+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:23.534949+0000","last_peered":"2024-09-10T20:32:23.534949+0000","last_clean":"2024-09-10T20:32:23.534949+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:23.534949+0000","last_undegraded":"2024-09-10T20:32:23.534949+0000","last_fullsized":"2024-09-10T20:32:23.534949+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16266,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534736+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:23.534736+0000","last_peered":"2024-09-10T20:32:23.534736+0000","last_clean":"2024-09-10T20:32:23.534736+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:23.534736+0000","last_undegraded":"2024-09-10T20:32:23.534736+0000","last_fullsized":"2024-09-10T20:32:23.534736+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535142+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:23.535142+0000","last_peered":"2024-09-10T20:32:23.535142+0000","last_clean":"2024-09-10T20:32:23.535142+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:23.535142+0000","last_undegraded":"2024-09-10T20:32:23.535142+0000","last_fullsized":"2024-09-10T20:32:23.535142+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535223+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:23.535223+0000","last_peered":"2024-09-10T20:32:23.535223+0000","last_clean":"2024-09-10T20:32:23.535223+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:23.535223+0000","last_undegraded":"2024-09-10T20:32:23.535223+0000","last_fullsized":"2024-09-10T20:32:23.535223+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:23.535061+0000","last_peered":"2024-09-10T20:32:23.535061+0000","last_clean":"2024-09-10T20:32:23.535061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:23.535061+0000","last_undegraded":"2024-09-10T20:32:23.535061+0000","last_fullsized":"2024-09-10T20:32:23.535061+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16159,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534230+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:23.534230+0000","last_peered":"2024-09-10T20:32:23.534230+0000","last_clean":"2024-09-10T20:32:23.534230+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:23.534230+0000","last_undegraded":"2024-09-10T20:32:23.534230+0000","last_fullsized":"2024-09-10T20:32:23.534230+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16166,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:24.954031+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:24.954031+0000","last_peered":"2024-09-10T20:32:24.954031+0000","last_clean":"2024-09-10T20:32:24.954031+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:24.954031+0000","last_undegraded":"2024-09-10T20:32:24.954031+0000","last_fullsized":"2024-09-10T20:32:24.954031+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12507,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535394+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:23.535394+0000","last_peered":"2024-09-10T20:32:23.535394+0000","last_clean":"2024-09-10T20:32:23.535394+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:23.535394+0000","last_undegraded":"2024-09-10T20:32:23.535394+0000","last_fullsized":"2024-09-10T20:32:23.535394+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12062,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535054+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:23.535054+0000","last_peered":"2024-09-10T20:32:23.535054+0000","last_clean":"2024-09-10T20:32:23.535054+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:23.535054+0000","last_undegraded":"2024-09-10T20:32:23.535054+0000","last_fullsized":"2024-09-10T20:32:23.535054+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535032+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:23.535032+0000","last_peered":"2024-09-10T20:32:23.535032+0000","last_clean":"2024-09-10T20:32:23.535032+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:23.535032+0000","last_undegraded":"2024-09-10T20:32:23.535032+0000","last_fullsized":"2024-09-10T20:32:23.535032+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12713,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535159+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:23.535159+0000","last_peered":"2024-09-10T20:32:23.535159+0000","last_clean":"2024-09-10T20:32:23.535159+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:23.535159+0000","last_undegraded":"2024-09-10T20:32:23.535159+0000","last_fullsized":"2024-09-10T20:32:23.535159+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16265,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535257+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:23.535257+0000","last_peered":"2024-09-10T20:32:23.535257+0000","last_clean":"2024-09-10T20:32:23.535257+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:23.535257+0000","last_undegraded":"2024-09-10T20:32:23.535257+0000","last_fullsized":"2024-09-10T20:32:23.535257+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699980,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671860,"statfs":{"total":96636764160,"available":95919984640,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703636,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668204,"statfs":{"total":96636764160,"available":95916240896,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600012,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771828,"statfs":{"total":96636764160,"available":96022351872,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:31.567 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:31.865 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:31.876 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-10T20:32:30.156436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695232,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792128,"statfs":{"total":386547056640,"available":383787139072,"internally_reserved":0,"allocated":2080768,"data_stored":796224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12107,"num_write_kb":-48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001272"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":13447,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:25.780757+0000","last_change":"2024-09-10T20:32:21.781527+0000","last_active":"2024-09-10T20:32:25.780757+0000","last_peered":"2024-09-10T20:32:25.780757+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:25.780757+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.536405+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:23.536405+0000","last_peered":"2024-09-10T20:32:23.536405+0000","last_clean":"2024-09-10T20:32:23.536405+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:23.536405+0000","last_undegraded":"2024-09-10T20:32:23.536405+0000","last_fullsized":"2024-09-10T20:32:23.536405+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16225,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534127+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:23.534127+0000","last_peered":"2024-09-10T20:32:23.534127+0000","last_clean":"2024-09-10T20:32:23.534127+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:23.534127+0000","last_undegraded":"2024-09-10T20:32:23.534127+0000","last_fullsized":"2024-09-10T20:32:23.534127+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16622,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534949+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:23.534949+0000","last_peered":"2024-09-10T20:32:23.534949+0000","last_clean":"2024-09-10T20:32:23.534949+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:23.534949+0000","last_undegraded":"2024-09-10T20:32:23.534949+0000","last_fullsized":"2024-09-10T20:32:23.534949+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16266,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534736+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:23.534736+0000","last_peered":"2024-09-10T20:32:23.534736+0000","last_clean":"2024-09-10T20:32:23.534736+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:23.534736+0000","last_undegraded":"2024-09-10T20:32:23.534736+0000","last_fullsized":"2024-09-10T20:32:23.534736+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535142+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:23.535142+0000","last_peered":"2024-09-10T20:32:23.535142+0000","last_clean":"2024-09-10T20:32:23.535142+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:23.535142+0000","last_undegraded":"2024-09-10T20:32:23.535142+0000","last_fullsized":"2024-09-10T20:32:23.535142+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535223+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:23.535223+0000","last_peered":"2024-09-10T20:32:23.535223+0000","last_clean":"2024-09-10T20:32:23.535223+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:23.535223+0000","last_undegraded":"2024-09-10T20:32:23.535223+0000","last_fullsized":"2024-09-10T20:32:23.535223+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:23.535061+0000","last_peered":"2024-09-10T20:32:23.535061+0000","last_clean":"2024-09-10T20:32:23.535061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:23.535061+0000","last_undegraded":"2024-09-10T20:32:23.535061+0000","last_fullsized":"2024-09-10T20:32:23.535061+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16159,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534230+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:23.534230+0000","last_peered":"2024-09-10T20:32:23.534230+0000","last_clean":"2024-09-10T20:32:23.534230+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:23.534230+0000","last_undegraded":"2024-09-10T20:32:23.534230+0000","last_fullsized":"2024-09-10T20:32:23.534230+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16166,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:24.954031+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:24.954031+0000","last_peered":"2024-09-10T20:32:24.954031+0000","last_clean":"2024-09-10T20:32:24.954031+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:24.954031+0000","last_undegraded":"2024-09-10T20:32:24.954031+0000","last_fullsized":"2024-09-10T20:32:24.954031+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12507,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535394+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:23.535394+0000","last_peered":"2024-09-10T20:32:23.535394+0000","last_clean":"2024-09-10T20:32:23.535394+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:23.535394+0000","last_undegraded":"2024-09-10T20:32:23.535394+0000","last_fullsized":"2024-09-10T20:32:23.535394+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12062,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535054+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:23.535054+0000","last_peered":"2024-09-10T20:32:23.535054+0000","last_clean":"2024-09-10T20:32:23.535054+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:23.535054+0000","last_undegraded":"2024-09-10T20:32:23.535054+0000","last_fullsized":"2024-09-10T20:32:23.535054+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535032+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:23.535032+0000","last_peered":"2024-09-10T20:32:23.535032+0000","last_clean":"2024-09-10T20:32:23.535032+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:23.535032+0000","last_undegraded":"2024-09-10T20:32:23.535032+0000","last_fullsized":"2024-09-10T20:32:23.535032+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12713,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535159+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:23.535159+0000","last_peered":"2024-09-10T20:32:23.535159+0000","last_clean":"2024-09-10T20:32:23.535159+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:23.535159+0000","last_undegraded":"2024-09-10T20:32:23.535159+0000","last_fullsized":"2024-09-10T20:32:23.535159+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16265,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535257+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:23.535257+0000","last_peered":"2024-09-10T20:32:23.535257+0000","last_clean":"2024-09-10T20:32:23.535257+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:23.535257+0000","last_undegraded":"2024-09-10T20:32:23.535257+0000","last_fullsized":"2024-09-10T20:32:23.535257+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699980,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671860,"statfs":{"total":96636764160,"available":95919984640,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703636,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668204,"statfs":{"total":96636764160,"available":95916240896,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587532,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784308,"statfs":{"total":96636764160,"available":96035131392,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:31.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:32.178 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:32.179 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:32.190 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-10T20:32:32.156761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3484,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695232,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792128,"statfs":{"total":386547056640,"available":383787139072,"internally_reserved":0,"allocated":2080768,"data_stored":796224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12107,"num_write_kb":-48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.633807"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":15321,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:30.783863+0000","last_change":"2024-09-10T20:32:21.781527+0000","last_active":"2024-09-10T20:32:30.783863+0000","last_peered":"2024-09-10T20:32:30.783863+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:30.783863+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.536405+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:23.536405+0000","last_peered":"2024-09-10T20:32:23.536405+0000","last_clean":"2024-09-10T20:32:23.536405+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:23.536405+0000","last_undegraded":"2024-09-10T20:32:23.536405+0000","last_fullsized":"2024-09-10T20:32:23.536405+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16225,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534127+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:23.534127+0000","last_peered":"2024-09-10T20:32:23.534127+0000","last_clean":"2024-09-10T20:32:23.534127+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:23.534127+0000","last_undegraded":"2024-09-10T20:32:23.534127+0000","last_fullsized":"2024-09-10T20:32:23.534127+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16622,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534949+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:23.534949+0000","last_peered":"2024-09-10T20:32:23.534949+0000","last_clean":"2024-09-10T20:32:23.534949+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:23.534949+0000","last_undegraded":"2024-09-10T20:32:23.534949+0000","last_fullsized":"2024-09-10T20:32:23.534949+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16266,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534736+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:23.534736+0000","last_peered":"2024-09-10T20:32:23.534736+0000","last_clean":"2024-09-10T20:32:23.534736+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:23.534736+0000","last_undegraded":"2024-09-10T20:32:23.534736+0000","last_fullsized":"2024-09-10T20:32:23.534736+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535142+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:23.535142+0000","last_peered":"2024-09-10T20:32:23.535142+0000","last_clean":"2024-09-10T20:32:23.535142+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:23.535142+0000","last_undegraded":"2024-09-10T20:32:23.535142+0000","last_fullsized":"2024-09-10T20:32:23.535142+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535223+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:23.535223+0000","last_peered":"2024-09-10T20:32:23.535223+0000","last_clean":"2024-09-10T20:32:23.535223+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:23.535223+0000","last_undegraded":"2024-09-10T20:32:23.535223+0000","last_fullsized":"2024-09-10T20:32:23.535223+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:23.535061+0000","last_peered":"2024-09-10T20:32:23.535061+0000","last_clean":"2024-09-10T20:32:23.535061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:23.535061+0000","last_undegraded":"2024-09-10T20:32:23.535061+0000","last_fullsized":"2024-09-10T20:32:23.535061+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16159,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534230+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:23.534230+0000","last_peered":"2024-09-10T20:32:23.534230+0000","last_clean":"2024-09-10T20:32:23.534230+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:23.534230+0000","last_undegraded":"2024-09-10T20:32:23.534230+0000","last_fullsized":"2024-09-10T20:32:23.534230+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16166,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:24.954031+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:24.954031+0000","last_peered":"2024-09-10T20:32:24.954031+0000","last_clean":"2024-09-10T20:32:24.954031+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:24.954031+0000","last_undegraded":"2024-09-10T20:32:24.954031+0000","last_fullsized":"2024-09-10T20:32:24.954031+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12507,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535394+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:23.535394+0000","last_peered":"2024-09-10T20:32:23.535394+0000","last_clean":"2024-09-10T20:32:23.535394+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:23.535394+0000","last_undegraded":"2024-09-10T20:32:23.535394+0000","last_fullsized":"2024-09-10T20:32:23.535394+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12062,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535054+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:23.535054+0000","last_peered":"2024-09-10T20:32:23.535054+0000","last_clean":"2024-09-10T20:32:23.535054+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:23.535054+0000","last_undegraded":"2024-09-10T20:32:23.535054+0000","last_fullsized":"2024-09-10T20:32:23.535054+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535032+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:23.535032+0000","last_peered":"2024-09-10T20:32:23.535032+0000","last_clean":"2024-09-10T20:32:23.535032+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:23.535032+0000","last_undegraded":"2024-09-10T20:32:23.535032+0000","last_fullsized":"2024-09-10T20:32:23.535032+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12713,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535159+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:23.535159+0000","last_peered":"2024-09-10T20:32:23.535159+0000","last_clean":"2024-09-10T20:32:23.535159+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:23.535159+0000","last_undegraded":"2024-09-10T20:32:23.535159+0000","last_fullsized":"2024-09-10T20:32:23.535159+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16265,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535257+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:23.535257+0000","last_peered":"2024-09-10T20:32:23.535257+0000","last_clean":"2024-09-10T20:32:23.535257+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:23.535257+0000","last_undegraded":"2024-09-10T20:32:23.535257+0000","last_fullsized":"2024-09-10T20:32:23.535257+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3484,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699980,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671860,"statfs":{"total":96636764160,"available":95919984640,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703636,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668204,"statfs":{"total":96636764160,"available":95916240896,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587532,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784308,"statfs":{"total":96636764160,"available":96035131392,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:32.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-10T20:32:32.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "44'12122", 'reported_seq': 15321, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-10T20:32:30.783863+0000', 'last_change': '2024-09-10T20:32:21.781527+0000', 'last_active': '2024-09-10T20:32:30.783863+0000', 'last_peered': '2024-09-10T20:32:30.783863+0000', 'last_clean': '2024-09-10T20:18:54.933927+0000', 'last_became_active': '2024-09-10T20:20:16.142872+0000', 'last_became_peered': '2024-09-10T20:20:16.142872+0000', 'last_unstale': '2024-09-10T20:32:30.783863+0000', 'last_undegraded': '2024-09-10T20:20:15.504418+0000', 'last_fullsized': '2024-09-10T20:20:15.457369+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'last_clean_scrub_stamp': '2024-09-10T20:18:54.933927+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 420, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12122, 'num_write_kb': 48488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1546, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -420}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:32:32.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:32.549 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:32.549 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1357/4 objects degraded (33925.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":841,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":62,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1,"data_bytes":6736,"bytes_used":2759917568,"bytes_avail":383787139072,"bytes_total":386547056640,"degraded_objects":1357,"degraded_total":4,"degraded_ratio":339.25},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-09-10T20:32:12.153719+0000","services":{}},"progress_events":{}} 2024-09-10T20:32:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-10T20:32:32.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:32.841 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:32.852 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-10T20:32:32.156761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3484,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2695232,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374792128,"statfs":{"total":386547056640,"available":383787139072,"internally_reserved":0,"allocated":2080768,"data_stored":796224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2757740086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12107,"num_write_kb":-48436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.633807"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":15321,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:30.783863+0000","last_change":"2024-09-10T20:32:21.781527+0000","last_active":"2024-09-10T20:32:30.783863+0000","last_peered":"2024-09-10T20:32:30.783863+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:30.783863+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.536405+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:23.536405+0000","last_peered":"2024-09-10T20:32:23.536405+0000","last_clean":"2024-09-10T20:32:23.536405+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:23.536405+0000","last_undegraded":"2024-09-10T20:32:23.536405+0000","last_fullsized":"2024-09-10T20:32:23.536405+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16225,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534127+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:23.534127+0000","last_peered":"2024-09-10T20:32:23.534127+0000","last_clean":"2024-09-10T20:32:23.534127+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:23.534127+0000","last_undegraded":"2024-09-10T20:32:23.534127+0000","last_fullsized":"2024-09-10T20:32:23.534127+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16622,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534949+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:23.534949+0000","last_peered":"2024-09-10T20:32:23.534949+0000","last_clean":"2024-09-10T20:32:23.534949+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:23.534949+0000","last_undegraded":"2024-09-10T20:32:23.534949+0000","last_fullsized":"2024-09-10T20:32:23.534949+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16266,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534736+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:23.534736+0000","last_peered":"2024-09-10T20:32:23.534736+0000","last_clean":"2024-09-10T20:32:23.534736+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:23.534736+0000","last_undegraded":"2024-09-10T20:32:23.534736+0000","last_fullsized":"2024-09-10T20:32:23.534736+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535142+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:23.535142+0000","last_peered":"2024-09-10T20:32:23.535142+0000","last_clean":"2024-09-10T20:32:23.535142+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:23.535142+0000","last_undegraded":"2024-09-10T20:32:23.535142+0000","last_fullsized":"2024-09-10T20:32:23.535142+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535223+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:23.535223+0000","last_peered":"2024-09-10T20:32:23.535223+0000","last_clean":"2024-09-10T20:32:23.535223+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:23.535223+0000","last_undegraded":"2024-09-10T20:32:23.535223+0000","last_fullsized":"2024-09-10T20:32:23.535223+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:23.535061+0000","last_peered":"2024-09-10T20:32:23.535061+0000","last_clean":"2024-09-10T20:32:23.535061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:23.535061+0000","last_undegraded":"2024-09-10T20:32:23.535061+0000","last_fullsized":"2024-09-10T20:32:23.535061+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16159,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534230+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:23.534230+0000","last_peered":"2024-09-10T20:32:23.534230+0000","last_clean":"2024-09-10T20:32:23.534230+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:23.534230+0000","last_undegraded":"2024-09-10T20:32:23.534230+0000","last_fullsized":"2024-09-10T20:32:23.534230+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16166,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:24.954031+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:24.954031+0000","last_peered":"2024-09-10T20:32:24.954031+0000","last_clean":"2024-09-10T20:32:24.954031+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:24.954031+0000","last_undegraded":"2024-09-10T20:32:24.954031+0000","last_fullsized":"2024-09-10T20:32:24.954031+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12507,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535394+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:23.535394+0000","last_peered":"2024-09-10T20:32:23.535394+0000","last_clean":"2024-09-10T20:32:23.535394+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:23.535394+0000","last_undegraded":"2024-09-10T20:32:23.535394+0000","last_fullsized":"2024-09-10T20:32:23.535394+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12062,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535054+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:23.535054+0000","last_peered":"2024-09-10T20:32:23.535054+0000","last_clean":"2024-09-10T20:32:23.535054+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:23.535054+0000","last_undegraded":"2024-09-10T20:32:23.535054+0000","last_fullsized":"2024-09-10T20:32:23.535054+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535032+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:23.535032+0000","last_peered":"2024-09-10T20:32:23.535032+0000","last_clean":"2024-09-10T20:32:23.535032+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:23.535032+0000","last_undegraded":"2024-09-10T20:32:23.535032+0000","last_fullsized":"2024-09-10T20:32:23.535032+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12713,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535159+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:23.535159+0000","last_peered":"2024-09-10T20:32:23.535159+0000","last_clean":"2024-09-10T20:32:23.535159+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:23.535159+0000","last_undegraded":"2024-09-10T20:32:23.535159+0000","last_fullsized":"2024-09-10T20:32:23.535159+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16265,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535257+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:23.535257+0000","last_peered":"2024-09-10T20:32:23.535257+0000","last_clean":"2024-09-10T20:32:23.535257+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:23.535257+0000","last_undegraded":"2024-09-10T20:32:23.535257+0000","last_fullsized":"2024-09-10T20:32:23.535257+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3484,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699980,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671860,"statfs":{"total":96636764160,"available":95919984640,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703636,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668204,"statfs":{"total":96636764160,"available":95916240896,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587532,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784308,"statfs":{"total":96636764160,"available":96035131392,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:35.855 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:36.140 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:36.141 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:36.152 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-10T20:32:35.559049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3484,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703552,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783808,"statfs":{"total":386547056640,"available":383778619392,"internally_reserved":0,"allocated":2080768,"data_stored":796224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":937,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.403275"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":15321,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-10T20:32:30.783863+0000","last_change":"2024-09-10T20:32:21.781527+0000","last_active":"2024-09-10T20:32:30.783863+0000","last_peered":"2024-09-10T20:32:30.783863+0000","last_clean":"2024-09-10T20:18:54.933927+0000","last_became_active":"2024-09-10T20:20:16.142872+0000","last_became_peered":"2024-09-10T20:20:16.142872+0000","last_unstale":"2024-09-10T20:32:30.783863+0000","last_undegraded":"2024-09-10T20:20:15.504418+0000","last_fullsized":"2024-09-10T20:20:15.457369+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12432,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557737+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:34.557737+0000","last_peered":"2024-09-10T20:32:34.557737+0000","last_clean":"2024-09-10T20:32:34.557737+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:34.557737+0000","last_undegraded":"2024-09-10T20:32:34.557737+0000","last_fullsized":"2024-09-10T20:32:34.557737+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16226,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534179+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:23.534179+0000","last_peered":"2024-09-10T20:32:23.534179+0000","last_clean":"2024-09-10T20:32:23.534179+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:23.534179+0000","last_undegraded":"2024-09-10T20:32:23.534179+0000","last_fullsized":"2024-09-10T20:32:23.534179+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16624,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557355+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:34.557355+0000","last_peered":"2024-09-10T20:32:34.557355+0000","last_clean":"2024-09-10T20:32:34.557355+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:34.557355+0000","last_undegraded":"2024-09-10T20:32:34.557355+0000","last_fullsized":"2024-09-10T20:32:34.557355+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16266,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534736+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:23.534736+0000","last_peered":"2024-09-10T20:32:23.534736+0000","last_clean":"2024-09-10T20:32:23.534736+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:23.534736+0000","last_undegraded":"2024-09-10T20:32:23.534736+0000","last_fullsized":"2024-09-10T20:32:23.534736+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535142+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:23.535142+0000","last_peered":"2024-09-10T20:32:23.535142+0000","last_clean":"2024-09-10T20:32:23.535142+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:23.535142+0000","last_undegraded":"2024-09-10T20:32:23.535142+0000","last_fullsized":"2024-09-10T20:32:23.535142+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12518,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557474+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:34.557474+0000","last_peered":"2024-09-10T20:32:34.557474+0000","last_clean":"2024-09-10T20:32:34.557474+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:34.557474+0000","last_undegraded":"2024-09-10T20:32:34.557474+0000","last_fullsized":"2024-09-10T20:32:34.557474+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.535061+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:23.535061+0000","last_peered":"2024-09-10T20:32:23.535061+0000","last_clean":"2024-09-10T20:32:23.535061+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:23.535061+0000","last_undegraded":"2024-09-10T20:32:23.535061+0000","last_fullsized":"2024-09-10T20:32:23.535061+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16159,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534230+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:23.534230+0000","last_peered":"2024-09-10T20:32:23.534230+0000","last_clean":"2024-09-10T20:32:23.534230+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:23.534230+0000","last_undegraded":"2024-09-10T20:32:23.534230+0000","last_fullsized":"2024-09-10T20:32:23.534230+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16166,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:24.954031+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:24.954031+0000","last_peered":"2024-09-10T20:32:24.954031+0000","last_clean":"2024-09-10T20:32:24.954031+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:24.954031+0000","last_undegraded":"2024-09-10T20:32:24.954031+0000","last_fullsized":"2024-09-10T20:32:24.954031+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12509,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557683+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:34.557683+0000","last_peered":"2024-09-10T20:32:34.557683+0000","last_clean":"2024-09-10T20:32:34.557683+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:34.557683+0000","last_undegraded":"2024-09-10T20:32:34.557683+0000","last_fullsized":"2024-09-10T20:32:34.557683+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12064,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557400+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:34.557400+0000","last_peered":"2024-09-10T20:32:34.557400+0000","last_clean":"2024-09-10T20:32:34.557400+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:34.557400+0000","last_undegraded":"2024-09-10T20:32:34.557400+0000","last_fullsized":"2024-09-10T20:32:34.557400+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16191,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557346+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:34.557346+0000","last_peered":"2024-09-10T20:32:34.557346+0000","last_clean":"2024-09-10T20:32:34.557346+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:34.557346+0000","last_undegraded":"2024-09-10T20:32:34.557346+0000","last_fullsized":"2024-09-10T20:32:34.557346+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12715,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557353+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:34.557353+0000","last_peered":"2024-09-10T20:32:34.557353+0000","last_clean":"2024-09-10T20:32:34.557353+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:34.557353+0000","last_undegraded":"2024-09-10T20:32:34.557353+0000","last_fullsized":"2024-09-10T20:32:34.557353+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16267,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557492+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:34.557492+0000","last_peered":"2024-09-10T20:32:34.557492+0000","last_clean":"2024-09-10T20:32:34.557492+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:34.557492+0000","last_undegraded":"2024-09-10T20:32:34.557492+0000","last_fullsized":"2024-09-10T20:32:34.557492+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3484,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699980,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671860,"statfs":{"total":96636764160,"available":95919984640,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703636,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668204,"statfs":{"total":96636764160,"available":95916240896,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595852,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775988,"statfs":{"total":96636764160,"available":96026611712,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:36.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:36.433 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:36.434 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:36.445 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-10T20:32:36.157388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703576,"kb_used_data":2048,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783784,"statfs":{"total":386547056640,"available":383778594816,"internally_reserved":0,"allocated":2097152,"data_stored":802339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1357,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001297"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12432,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557737+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:34.557737+0000","last_peered":"2024-09-10T20:32:34.557737+0000","last_clean":"2024-09-10T20:32:34.557737+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:34.557737+0000","last_undegraded":"2024-09-10T20:32:34.557737+0000","last_fullsized":"2024-09-10T20:32:34.557737+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16226,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534179+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:23.534179+0000","last_peered":"2024-09-10T20:32:23.534179+0000","last_clean":"2024-09-10T20:32:23.534179+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:23.534179+0000","last_undegraded":"2024-09-10T20:32:23.534179+0000","last_fullsized":"2024-09-10T20:32:23.534179+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16624,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557355+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:34.557355+0000","last_peered":"2024-09-10T20:32:34.557355+0000","last_clean":"2024-09-10T20:32:34.557355+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:34.557355+0000","last_undegraded":"2024-09-10T20:32:34.557355+0000","last_fullsized":"2024-09-10T20:32:34.557355+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12518,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557474+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:34.557474+0000","last_peered":"2024-09-10T20:32:34.557474+0000","last_clean":"2024-09-10T20:32:34.557474+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:34.557474+0000","last_undegraded":"2024-09-10T20:32:34.557474+0000","last_fullsized":"2024-09-10T20:32:34.557474+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16159,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534230+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:23.534230+0000","last_peered":"2024-09-10T20:32:23.534230+0000","last_clean":"2024-09-10T20:32:23.534230+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:23.534230+0000","last_undegraded":"2024-09-10T20:32:23.534230+0000","last_fullsized":"2024-09-10T20:32:23.534230+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12509,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557683+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:34.557683+0000","last_peered":"2024-09-10T20:32:34.557683+0000","last_clean":"2024-09-10T20:32:34.557683+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:34.557683+0000","last_undegraded":"2024-09-10T20:32:34.557683+0000","last_fullsized":"2024-09-10T20:32:34.557683+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12064,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557400+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:34.557400+0000","last_peered":"2024-09-10T20:32:34.557400+0000","last_clean":"2024-09-10T20:32:34.557400+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:34.557400+0000","last_undegraded":"2024-09-10T20:32:34.557400+0000","last_fullsized":"2024-09-10T20:32:34.557400+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16191,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557346+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:34.557346+0000","last_peered":"2024-09-10T20:32:34.557346+0000","last_clean":"2024-09-10T20:32:34.557346+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:34.557346+0000","last_undegraded":"2024-09-10T20:32:34.557346+0000","last_fullsized":"2024-09-10T20:32:34.557346+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12715,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557353+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:34.557353+0000","last_peered":"2024-09-10T20:32:34.557353+0000","last_clean":"2024-09-10T20:32:34.557353+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:34.557353+0000","last_undegraded":"2024-09-10T20:32:34.557353+0000","last_fullsized":"2024-09-10T20:32:34.557353+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16267,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557492+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:34.557492+0000","last_peered":"2024-09-10T20:32:34.557492+0000","last_clean":"2024-09-10T20:32:34.557492+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:34.557492+0000","last_undegraded":"2024-09-10T20:32:34.557492+0000","last_fullsized":"2024-09-10T20:32:34.557492+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703636,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668204,"statfs":{"total":96636764160,"available":95916240896,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595852,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775988,"statfs":{"total":96636764160,"available":96026611712,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:36.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:32:36.796 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:36.797 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":845,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":64,"num_osds":4,"num_up_osds":4,"osd_up_since":1725999613,"num_in_osds":4,"osd_in_since":1725999512,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16}],"num_pgs":16,"num_pools":1,"num_objects":1,"data_bytes":6736,"bytes_used":2768461824,"bytes_avail":383778594816,"bytes_total":386547056640,"recovering_objects_per_sec":169,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1357,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-10T20:18:31:254942+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-09-10T20:32:12.153719+0000","services":{}},"progress_events":{}} 2024-09-10T20:32:36.809 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-10T20:32:36.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:37.092 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:37.103 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-10T20:32:36.157388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703576,"kb_used_data":2048,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783784,"statfs":{"total":386547056640,"available":383778594816,"internally_reserved":0,"allocated":2097152,"data_stored":802339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1357,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001297"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12432,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557737+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:34.557737+0000","last_peered":"2024-09-10T20:32:34.557737+0000","last_clean":"2024-09-10T20:32:34.557737+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:34.557737+0000","last_undegraded":"2024-09-10T20:32:34.557737+0000","last_fullsized":"2024-09-10T20:32:34.557737+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16226,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534179+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:23.534179+0000","last_peered":"2024-09-10T20:32:23.534179+0000","last_clean":"2024-09-10T20:32:23.534179+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:23.534179+0000","last_undegraded":"2024-09-10T20:32:23.534179+0000","last_fullsized":"2024-09-10T20:32:23.534179+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16624,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557355+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:34.557355+0000","last_peered":"2024-09-10T20:32:34.557355+0000","last_clean":"2024-09-10T20:32:34.557355+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:34.557355+0000","last_undegraded":"2024-09-10T20:32:34.557355+0000","last_fullsized":"2024-09-10T20:32:34.557355+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16126,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.532800+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:23.532800+0000","last_peered":"2024-09-10T20:32:23.532800+0000","last_clean":"2024-09-10T20:32:23.532800+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:23.532800+0000","last_undegraded":"2024-09-10T20:32:23.532800+0000","last_fullsized":"2024-09-10T20:32:23.532800+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12518,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557474+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:34.557474+0000","last_peered":"2024-09-10T20:32:34.557474+0000","last_clean":"2024-09-10T20:32:34.557474+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:34.557474+0000","last_undegraded":"2024-09-10T20:32:34.557474+0000","last_fullsized":"2024-09-10T20:32:34.557474+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16159,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-10T20:32:23.534230+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:23.534230+0000","last_peered":"2024-09-10T20:32:23.534230+0000","last_clean":"2024-09-10T20:32:23.534230+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:23.534230+0000","last_undegraded":"2024-09-10T20:32:23.534230+0000","last_fullsized":"2024-09-10T20:32:23.534230+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12509,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557683+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:34.557683+0000","last_peered":"2024-09-10T20:32:34.557683+0000","last_clean":"2024-09-10T20:32:34.557683+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:34.557683+0000","last_undegraded":"2024-09-10T20:32:34.557683+0000","last_fullsized":"2024-09-10T20:32:34.557683+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12064,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557400+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:34.557400+0000","last_peered":"2024-09-10T20:32:34.557400+0000","last_clean":"2024-09-10T20:32:34.557400+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:34.557400+0000","last_undegraded":"2024-09-10T20:32:34.557400+0000","last_fullsized":"2024-09-10T20:32:34.557400+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16191,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557346+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:34.557346+0000","last_peered":"2024-09-10T20:32:34.557346+0000","last_clean":"2024-09-10T20:32:34.557346+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:34.557346+0000","last_undegraded":"2024-09-10T20:32:34.557346+0000","last_fullsized":"2024-09-10T20:32:34.557346+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12715,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557353+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:34.557353+0000","last_peered":"2024-09-10T20:32:34.557353+0000","last_clean":"2024-09-10T20:32:34.557353+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:34.557353+0000","last_undegraded":"2024-09-10T20:32:34.557353+0000","last_fullsized":"2024-09-10T20:32:34.557353+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16267,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-10T20:32:34.557492+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:34.557492+0000","last_peered":"2024-09-10T20:32:34.557492+0000","last_clean":"2024-09-10T20:32:34.557492+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:34.557492+0000","last_undegraded":"2024-09-10T20:32:34.557492+0000","last_fullsized":"2024-09-10T20:32:34.557492+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703636,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668204,"statfs":{"total":96636764160,"available":95916240896,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595852,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775988,"statfs":{"total":96636764160,"available":96026611712,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704084,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667756,"statfs":{"total":96636764160,"available":95915782144,"internally_reserved":0,"allocated":520192,"data_stored":199056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:40.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:40.398 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:40.399 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:40.410 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-10T20:32:40.158136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703632,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783728,"statfs":{"total":386547056640,"available":383778537472,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001374"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12434,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562408+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:35.562408+0000","last_peered":"2024-09-10T20:32:35.562408+0000","last_clean":"2024-09-10T20:32:35.562408+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:35.562408+0000","last_undegraded":"2024-09-10T20:32:35.562408+0000","last_fullsized":"2024-09-10T20:32:35.562408+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.560571+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:35.560571+0000","last_peered":"2024-09-10T20:32:35.560571+0000","last_clean":"2024-09-10T20:32:35.560571+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:35.560571+0000","last_undegraded":"2024-09-10T20:32:35.560571+0000","last_fullsized":"2024-09-10T20:32:35.560571+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16626,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562227+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:35.562227+0000","last_peered":"2024-09-10T20:32:35.562227+0000","last_clean":"2024-09-10T20:32:35.562227+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:35.562227+0000","last_undegraded":"2024-09-10T20:32:35.562227+0000","last_fullsized":"2024-09-10T20:32:35.562227+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16130,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561475+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:35.561475+0000","last_peered":"2024-09-10T20:32:35.561475+0000","last_clean":"2024-09-10T20:32:35.561475+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:35.561475+0000","last_undegraded":"2024-09-10T20:32:35.561475+0000","last_fullsized":"2024-09-10T20:32:35.561475+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12520,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562015+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:35.562015+0000","last_peered":"2024-09-10T20:32:35.562015+0000","last_clean":"2024-09-10T20:32:35.562015+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:35.562015+0000","last_undegraded":"2024-09-10T20:32:35.562015+0000","last_fullsized":"2024-09-10T20:32:35.562015+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561227+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:35.561227+0000","last_peered":"2024-09-10T20:32:35.561227+0000","last_clean":"2024-09-10T20:32:35.561227+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:35.561227+0000","last_undegraded":"2024-09-10T20:32:35.561227+0000","last_fullsized":"2024-09-10T20:32:35.561227+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12511,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562231+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:35.562231+0000","last_peered":"2024-09-10T20:32:35.562231+0000","last_clean":"2024-09-10T20:32:35.562231+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:35.562231+0000","last_undegraded":"2024-09-10T20:32:35.562231+0000","last_fullsized":"2024-09-10T20:32:35.562231+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12066,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562042+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:35.562042+0000","last_peered":"2024-09-10T20:32:35.562042+0000","last_clean":"2024-09-10T20:32:35.562042+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:35.562042+0000","last_undegraded":"2024-09-10T20:32:35.562042+0000","last_fullsized":"2024-09-10T20:32:35.562042+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16193,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562247+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:35.562247+0000","last_peered":"2024-09-10T20:32:35.562247+0000","last_clean":"2024-09-10T20:32:35.562247+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:35.562247+0000","last_undegraded":"2024-09-10T20:32:35.562247+0000","last_fullsized":"2024-09-10T20:32:35.562247+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12717,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562118+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:35.562118+0000","last_peered":"2024-09-10T20:32:35.562118+0000","last_clean":"2024-09-10T20:32:35.562118+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:35.562118+0000","last_undegraded":"2024-09-10T20:32:35.562118+0000","last_fullsized":"2024-09-10T20:32:35.562118+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16269,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562140+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:35.562140+0000","last_peered":"2024-09-10T20:32:35.562140+0000","last_clean":"2024-09-10T20:32:35.562140+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:35.562140+0000","last_undegraded":"2024-09-10T20:32:35.562140+0000","last_fullsized":"2024-09-10T20:32:35.562140+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703652,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668188,"statfs":{"total":96636764160,"available":95916224512,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:40.411 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-10T20:32:40.412 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-10T20:32:40.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:40.693 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:40.693 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:40.705 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-10T20:32:40.158136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703632,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783728,"statfs":{"total":386547056640,"available":383778537472,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001374"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12434,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562408+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:35.562408+0000","last_peered":"2024-09-10T20:32:35.562408+0000","last_clean":"2024-09-10T20:32:35.562408+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:35.562408+0000","last_undegraded":"2024-09-10T20:32:35.562408+0000","last_fullsized":"2024-09-10T20:32:35.562408+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.560571+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:35.560571+0000","last_peered":"2024-09-10T20:32:35.560571+0000","last_clean":"2024-09-10T20:32:35.560571+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:35.560571+0000","last_undegraded":"2024-09-10T20:32:35.560571+0000","last_fullsized":"2024-09-10T20:32:35.560571+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16626,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562227+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:35.562227+0000","last_peered":"2024-09-10T20:32:35.562227+0000","last_clean":"2024-09-10T20:32:35.562227+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:35.562227+0000","last_undegraded":"2024-09-10T20:32:35.562227+0000","last_fullsized":"2024-09-10T20:32:35.562227+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16130,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561475+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:35.561475+0000","last_peered":"2024-09-10T20:32:35.561475+0000","last_clean":"2024-09-10T20:32:35.561475+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:35.561475+0000","last_undegraded":"2024-09-10T20:32:35.561475+0000","last_fullsized":"2024-09-10T20:32:35.561475+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12520,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562015+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:35.562015+0000","last_peered":"2024-09-10T20:32:35.562015+0000","last_clean":"2024-09-10T20:32:35.562015+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:35.562015+0000","last_undegraded":"2024-09-10T20:32:35.562015+0000","last_fullsized":"2024-09-10T20:32:35.562015+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561227+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:35.561227+0000","last_peered":"2024-09-10T20:32:35.561227+0000","last_clean":"2024-09-10T20:32:35.561227+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:35.561227+0000","last_undegraded":"2024-09-10T20:32:35.561227+0000","last_fullsized":"2024-09-10T20:32:35.561227+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12511,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562231+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:35.562231+0000","last_peered":"2024-09-10T20:32:35.562231+0000","last_clean":"2024-09-10T20:32:35.562231+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:35.562231+0000","last_undegraded":"2024-09-10T20:32:35.562231+0000","last_fullsized":"2024-09-10T20:32:35.562231+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12066,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562042+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:35.562042+0000","last_peered":"2024-09-10T20:32:35.562042+0000","last_clean":"2024-09-10T20:32:35.562042+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:35.562042+0000","last_undegraded":"2024-09-10T20:32:35.562042+0000","last_fullsized":"2024-09-10T20:32:35.562042+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16193,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562247+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:35.562247+0000","last_peered":"2024-09-10T20:32:35.562247+0000","last_clean":"2024-09-10T20:32:35.562247+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:35.562247+0000","last_undegraded":"2024-09-10T20:32:35.562247+0000","last_fullsized":"2024-09-10T20:32:35.562247+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12717,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562118+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:35.562118+0000","last_peered":"2024-09-10T20:32:35.562118+0000","last_clean":"2024-09-10T20:32:35.562118+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:35.562118+0000","last_undegraded":"2024-09-10T20:32:35.562118+0000","last_fullsized":"2024-09-10T20:32:35.562118+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16269,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562140+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:35.562140+0000","last_peered":"2024-09-10T20:32:35.562140+0000","last_clean":"2024-09-10T20:32:35.562140+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:35.562140+0000","last_undegraded":"2024-09-10T20:32:35.562140+0000","last_fullsized":"2024-09-10T20:32:35.562140+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703652,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668188,"statfs":{"total":96636764160,"available":95916224512,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:40.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:40.983 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:40.984 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:40.996 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-10T20:32:40.158136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703632,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783728,"statfs":{"total":386547056640,"available":383778537472,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001374"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12434,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562408+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:35.562408+0000","last_peered":"2024-09-10T20:32:35.562408+0000","last_clean":"2024-09-10T20:32:35.562408+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:35.562408+0000","last_undegraded":"2024-09-10T20:32:35.562408+0000","last_fullsized":"2024-09-10T20:32:35.562408+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.560571+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:35.560571+0000","last_peered":"2024-09-10T20:32:35.560571+0000","last_clean":"2024-09-10T20:32:35.560571+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:35.560571+0000","last_undegraded":"2024-09-10T20:32:35.560571+0000","last_fullsized":"2024-09-10T20:32:35.560571+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16626,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562227+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:35.562227+0000","last_peered":"2024-09-10T20:32:35.562227+0000","last_clean":"2024-09-10T20:32:35.562227+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:35.562227+0000","last_undegraded":"2024-09-10T20:32:35.562227+0000","last_fullsized":"2024-09-10T20:32:35.562227+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16130,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561475+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:35.561475+0000","last_peered":"2024-09-10T20:32:35.561475+0000","last_clean":"2024-09-10T20:32:35.561475+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:35.561475+0000","last_undegraded":"2024-09-10T20:32:35.561475+0000","last_fullsized":"2024-09-10T20:32:35.561475+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12520,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562015+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:35.562015+0000","last_peered":"2024-09-10T20:32:35.562015+0000","last_clean":"2024-09-10T20:32:35.562015+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:35.562015+0000","last_undegraded":"2024-09-10T20:32:35.562015+0000","last_fullsized":"2024-09-10T20:32:35.562015+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561227+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:35.561227+0000","last_peered":"2024-09-10T20:32:35.561227+0000","last_clean":"2024-09-10T20:32:35.561227+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:35.561227+0000","last_undegraded":"2024-09-10T20:32:35.561227+0000","last_fullsized":"2024-09-10T20:32:35.561227+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12511,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562231+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:35.562231+0000","last_peered":"2024-09-10T20:32:35.562231+0000","last_clean":"2024-09-10T20:32:35.562231+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:35.562231+0000","last_undegraded":"2024-09-10T20:32:35.562231+0000","last_fullsized":"2024-09-10T20:32:35.562231+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12066,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562042+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:35.562042+0000","last_peered":"2024-09-10T20:32:35.562042+0000","last_clean":"2024-09-10T20:32:35.562042+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:35.562042+0000","last_undegraded":"2024-09-10T20:32:35.562042+0000","last_fullsized":"2024-09-10T20:32:35.562042+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16193,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562247+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:35.562247+0000","last_peered":"2024-09-10T20:32:35.562247+0000","last_clean":"2024-09-10T20:32:35.562247+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:35.562247+0000","last_undegraded":"2024-09-10T20:32:35.562247+0000","last_fullsized":"2024-09-10T20:32:35.562247+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12717,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562118+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:35.562118+0000","last_peered":"2024-09-10T20:32:35.562118+0000","last_clean":"2024-09-10T20:32:35.562118+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:35.562118+0000","last_undegraded":"2024-09-10T20:32:35.562118+0000","last_fullsized":"2024-09-10T20:32:35.562118+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16269,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562140+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:35.562140+0000","last_peered":"2024-09-10T20:32:35.562140+0000","last_clean":"2024-09-10T20:32:35.562140+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:35.562140+0000","last_undegraded":"2024-09-10T20:32:35.562140+0000","last_fullsized":"2024-09-10T20:32:35.562140+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703652,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668188,"statfs":{"total":96636764160,"available":95916224512,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:40.997 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-10T20:32:40.997 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-10T20:32:41.009 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:32:41.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:41.333 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:41.334 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-10T20:32:40.158136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703632,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783728,"statfs":{"total":386547056640,"available":383778537472,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001374"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12434,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562408+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:35.562408+0000","last_peered":"2024-09-10T20:32:35.562408+0000","last_clean":"2024-09-10T20:32:35.562408+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:35.562408+0000","last_undegraded":"2024-09-10T20:32:35.562408+0000","last_fullsized":"2024-09-10T20:32:35.562408+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.560571+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:35.560571+0000","last_peered":"2024-09-10T20:32:35.560571+0000","last_clean":"2024-09-10T20:32:35.560571+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:35.560571+0000","last_undegraded":"2024-09-10T20:32:35.560571+0000","last_fullsized":"2024-09-10T20:32:35.560571+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16626,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562227+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:35.562227+0000","last_peered":"2024-09-10T20:32:35.562227+0000","last_clean":"2024-09-10T20:32:35.562227+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:35.562227+0000","last_undegraded":"2024-09-10T20:32:35.562227+0000","last_fullsized":"2024-09-10T20:32:35.562227+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16130,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561475+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:35.561475+0000","last_peered":"2024-09-10T20:32:35.561475+0000","last_clean":"2024-09-10T20:32:35.561475+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:35.561475+0000","last_undegraded":"2024-09-10T20:32:35.561475+0000","last_fullsized":"2024-09-10T20:32:35.561475+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12520,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562015+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:35.562015+0000","last_peered":"2024-09-10T20:32:35.562015+0000","last_clean":"2024-09-10T20:32:35.562015+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:35.562015+0000","last_undegraded":"2024-09-10T20:32:35.562015+0000","last_fullsized":"2024-09-10T20:32:35.562015+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561227+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:35.561227+0000","last_peered":"2024-09-10T20:32:35.561227+0000","last_clean":"2024-09-10T20:32:35.561227+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:35.561227+0000","last_undegraded":"2024-09-10T20:32:35.561227+0000","last_fullsized":"2024-09-10T20:32:35.561227+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12511,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562231+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:35.562231+0000","last_peered":"2024-09-10T20:32:35.562231+0000","last_clean":"2024-09-10T20:32:35.562231+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:35.562231+0000","last_undegraded":"2024-09-10T20:32:35.562231+0000","last_fullsized":"2024-09-10T20:32:35.562231+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12066,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562042+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:35.562042+0000","last_peered":"2024-09-10T20:32:35.562042+0000","last_clean":"2024-09-10T20:32:35.562042+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:35.562042+0000","last_undegraded":"2024-09-10T20:32:35.562042+0000","last_fullsized":"2024-09-10T20:32:35.562042+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16193,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562247+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:35.562247+0000","last_peered":"2024-09-10T20:32:35.562247+0000","last_clean":"2024-09-10T20:32:35.562247+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:35.562247+0000","last_undegraded":"2024-09-10T20:32:35.562247+0000","last_fullsized":"2024-09-10T20:32:35.562247+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12717,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562118+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:35.562118+0000","last_peered":"2024-09-10T20:32:35.562118+0000","last_clean":"2024-09-10T20:32:35.562118+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:35.562118+0000","last_undegraded":"2024-09-10T20:32:35.562118+0000","last_fullsized":"2024-09-10T20:32:35.562118+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16269,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562140+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:35.562140+0000","last_peered":"2024-09-10T20:32:35.562140+0000","last_clean":"2024-09-10T20:32:35.562140+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:35.562140+0000","last_undegraded":"2024-09-10T20:32:35.562140+0000","last_fullsized":"2024-09-10T20:32:35.562140+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703652,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668188,"statfs":{"total":96636764160,"available":95916224512,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:41.335 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:41.618 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:41.629 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-10T20:32:40.158136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703632,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783728,"statfs":{"total":386547056640,"available":383778537472,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001374"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12434,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562408+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:35.562408+0000","last_peered":"2024-09-10T20:32:35.562408+0000","last_clean":"2024-09-10T20:32:35.562408+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:35.562408+0000","last_undegraded":"2024-09-10T20:32:35.562408+0000","last_fullsized":"2024-09-10T20:32:35.562408+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.560571+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:35.560571+0000","last_peered":"2024-09-10T20:32:35.560571+0000","last_clean":"2024-09-10T20:32:35.560571+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:35.560571+0000","last_undegraded":"2024-09-10T20:32:35.560571+0000","last_fullsized":"2024-09-10T20:32:35.560571+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16626,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562227+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:35.562227+0000","last_peered":"2024-09-10T20:32:35.562227+0000","last_clean":"2024-09-10T20:32:35.562227+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:35.562227+0000","last_undegraded":"2024-09-10T20:32:35.562227+0000","last_fullsized":"2024-09-10T20:32:35.562227+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16130,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561475+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:35.561475+0000","last_peered":"2024-09-10T20:32:35.561475+0000","last_clean":"2024-09-10T20:32:35.561475+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:35.561475+0000","last_undegraded":"2024-09-10T20:32:35.561475+0000","last_fullsized":"2024-09-10T20:32:35.561475+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12520,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562015+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:35.562015+0000","last_peered":"2024-09-10T20:32:35.562015+0000","last_clean":"2024-09-10T20:32:35.562015+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:35.562015+0000","last_undegraded":"2024-09-10T20:32:35.562015+0000","last_fullsized":"2024-09-10T20:32:35.562015+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561227+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:35.561227+0000","last_peered":"2024-09-10T20:32:35.561227+0000","last_clean":"2024-09-10T20:32:35.561227+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:35.561227+0000","last_undegraded":"2024-09-10T20:32:35.561227+0000","last_fullsized":"2024-09-10T20:32:35.561227+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12511,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562231+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:35.562231+0000","last_peered":"2024-09-10T20:32:35.562231+0000","last_clean":"2024-09-10T20:32:35.562231+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:35.562231+0000","last_undegraded":"2024-09-10T20:32:35.562231+0000","last_fullsized":"2024-09-10T20:32:35.562231+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12066,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562042+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:35.562042+0000","last_peered":"2024-09-10T20:32:35.562042+0000","last_clean":"2024-09-10T20:32:35.562042+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:35.562042+0000","last_undegraded":"2024-09-10T20:32:35.562042+0000","last_fullsized":"2024-09-10T20:32:35.562042+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16193,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562247+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:35.562247+0000","last_peered":"2024-09-10T20:32:35.562247+0000","last_clean":"2024-09-10T20:32:35.562247+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:35.562247+0000","last_undegraded":"2024-09-10T20:32:35.562247+0000","last_fullsized":"2024-09-10T20:32:35.562247+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12717,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562118+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:35.562118+0000","last_peered":"2024-09-10T20:32:35.562118+0000","last_clean":"2024-09-10T20:32:35.562118+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:35.562118+0000","last_undegraded":"2024-09-10T20:32:35.562118+0000","last_fullsized":"2024-09-10T20:32:35.562118+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16269,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562140+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:35.562140+0000","last_peered":"2024-09-10T20:32:35.562140+0000","last_clean":"2024-09-10T20:32:35.562140+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:35.562140+0000","last_undegraded":"2024-09-10T20:32:35.562140+0000","last_fullsized":"2024-09-10T20:32:35.562140+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703652,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668188,"statfs":{"total":96636764160,"available":95916224512,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:41.630 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:32:41.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:41.909 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:41.921 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-10T20:32:40.158136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703632,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783728,"statfs":{"total":386547056640,"available":383778537472,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001374"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12434,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562408+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:35.562408+0000","last_peered":"2024-09-10T20:32:35.562408+0000","last_clean":"2024-09-10T20:32:35.562408+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:35.562408+0000","last_undegraded":"2024-09-10T20:32:35.562408+0000","last_fullsized":"2024-09-10T20:32:35.562408+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.560571+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:35.560571+0000","last_peered":"2024-09-10T20:32:35.560571+0000","last_clean":"2024-09-10T20:32:35.560571+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:35.560571+0000","last_undegraded":"2024-09-10T20:32:35.560571+0000","last_fullsized":"2024-09-10T20:32:35.560571+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16626,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562227+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:35.562227+0000","last_peered":"2024-09-10T20:32:35.562227+0000","last_clean":"2024-09-10T20:32:35.562227+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:35.562227+0000","last_undegraded":"2024-09-10T20:32:35.562227+0000","last_fullsized":"2024-09-10T20:32:35.562227+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16130,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561475+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:35.561475+0000","last_peered":"2024-09-10T20:32:35.561475+0000","last_clean":"2024-09-10T20:32:35.561475+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:35.561475+0000","last_undegraded":"2024-09-10T20:32:35.561475+0000","last_fullsized":"2024-09-10T20:32:35.561475+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12520,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562015+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:35.562015+0000","last_peered":"2024-09-10T20:32:35.562015+0000","last_clean":"2024-09-10T20:32:35.562015+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:35.562015+0000","last_undegraded":"2024-09-10T20:32:35.562015+0000","last_fullsized":"2024-09-10T20:32:35.562015+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561227+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:35.561227+0000","last_peered":"2024-09-10T20:32:35.561227+0000","last_clean":"2024-09-10T20:32:35.561227+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:35.561227+0000","last_undegraded":"2024-09-10T20:32:35.561227+0000","last_fullsized":"2024-09-10T20:32:35.561227+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12511,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562231+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:35.562231+0000","last_peered":"2024-09-10T20:32:35.562231+0000","last_clean":"2024-09-10T20:32:35.562231+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:35.562231+0000","last_undegraded":"2024-09-10T20:32:35.562231+0000","last_fullsized":"2024-09-10T20:32:35.562231+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12066,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562042+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:35.562042+0000","last_peered":"2024-09-10T20:32:35.562042+0000","last_clean":"2024-09-10T20:32:35.562042+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:35.562042+0000","last_undegraded":"2024-09-10T20:32:35.562042+0000","last_fullsized":"2024-09-10T20:32:35.562042+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16193,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562247+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:35.562247+0000","last_peered":"2024-09-10T20:32:35.562247+0000","last_clean":"2024-09-10T20:32:35.562247+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:35.562247+0000","last_undegraded":"2024-09-10T20:32:35.562247+0000","last_fullsized":"2024-09-10T20:32:35.562247+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12717,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562118+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:35.562118+0000","last_peered":"2024-09-10T20:32:35.562118+0000","last_clean":"2024-09-10T20:32:35.562118+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:35.562118+0000","last_undegraded":"2024-09-10T20:32:35.562118+0000","last_fullsized":"2024-09-10T20:32:35.562118+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16269,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562140+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:35.562140+0000","last_peered":"2024-09-10T20:32:35.562140+0000","last_clean":"2024-09-10T20:32:35.562140+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:35.562140+0000","last_undegraded":"2024-09-10T20:32:35.562140+0000","last_fullsized":"2024-09-10T20:32:35.562140+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700004,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671836,"statfs":{"total":96636764160,"available":95919960064,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703652,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668188,"statfs":{"total":96636764160,"available":95916224512,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:41.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:32:42.208 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:42.208 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":64,"fsid":"8fa38b67-7aca-4cf3-911d-119b3351a56d","created":"2024-09-10T20:18:31.255327+0000","modified":"2024-09-10T20:32:35.554496+0000","last_up_change":"2024-09-10T20:20:13.417830+0000","last_in_change":"2024-09-10T20:18:32.773408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-10T20:18:53.939346+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"7f22b5e1-0040-4c13-a24c-ed6610a42665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":48,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":3732379198}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":3732379198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":3732379198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":3732379198}]},"public_addr":"172.21.15.103:6800/3732379198","cluster_addr":"172.21.15.103:6801/3732379198","heartbeat_back_addr":"172.21.15.103:6803/3732379198","heartbeat_front_addr":"172.21.15.103:6802/3732379198","state":["exists","up"]},{"osd":1,"uuid":"dd79e308-70dc-48ff-9e8f-ff6225c2e718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":30,"up_thru":55,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":2304361244}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":2304361244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":2304361244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":2304361244}]},"public_addr":"172.21.15.103:6812/2304361244","cluster_addr":"172.21.15.103:6813/2304361244","heartbeat_back_addr":"172.21.15.103:6815/2304361244","heartbeat_front_addr":"172.21.15.103:6814/2304361244","state":["exists","up"]},{"osd":2,"uuid":"bca7fcd4-7b60-4f83-a11e-ca382ee1cc34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":3799088219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3799088219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3799088219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3799088219}]},"public_addr":"172.21.15.103:6804/3799088219","cluster_addr":"172.21.15.103:6805/3799088219","heartbeat_back_addr":"172.21.15.103:6807/3799088219","heartbeat_front_addr":"172.21.15.103:6806/3799088219","state":["exists","up"]},{"osd":3,"uuid":"a013b06c-c54b-4a97-892b-1b5dad559339","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":27,"up_thru":63,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":3618344502}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":3618344502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":3618344502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":3618344502}]},"public_addr":"172.21.15.103:6808/3618344502","cluster_addr":"172.21.15.103:6809/3618344502","heartbeat_back_addr":"172.21.15.103:6811/3618344502","heartbeat_front_addr":"172.21.15.103:6810/3618344502","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.124091+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.558785+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.498873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:18:39.872414+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:32:43.222 INFO:tasks.ceph:Scrubbing osd.0 2024-09-10T20:32:43.222 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:32:43.356 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:32:43.356 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:32:43.356 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:32:43.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-10T20:32:43.654 INFO:teuthology.orchestra.run.smithi103.stderr:instructed osd(s) 0 to deep-scrub 2024-09-10T20:32:43.667 INFO:tasks.ceph:Scrubbing osd.1 2024-09-10T20:32:43.667 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:32:43.789 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:32:43.789 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:32:43.789 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:32:43.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-10T20:32:44.081 INFO:teuthology.orchestra.run.smithi103.stderr:instructed osd(s) 1 to deep-scrub 2024-09-10T20:32:44.096 INFO:tasks.ceph:Scrubbing osd.2 2024-09-10T20:32:44.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:32:44.216 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:32:44.217 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:32:44.217 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:32:44.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-10T20:32:44.510 INFO:teuthology.orchestra.run.smithi103.stderr:instructed osd(s) 2 to deep-scrub 2024-09-10T20:32:44.524 INFO:tasks.ceph:Scrubbing osd.3 2024-09-10T20:32:44.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:32:44.646 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-10T20:32:44.647 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:32:44.647 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-10T20:32:44.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-10T20:32:44.940 INFO:teuthology.orchestra.run.smithi103.stderr:instructed osd(s) 3 to deep-scrub 2024-09-10T20:32:44.953 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:32:45.233 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:32:45.234 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:32:45.245 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-09-10T20:32:44.158744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703624,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783736,"statfs":{"total":386547056640,"available":383778545664,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.603767"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16174,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.630384+0000","last_change":"2024-09-10T20:32:35.630283+0000","last_active":"2024-09-10T20:32:35.630384+0000","last_peered":"2024-09-10T20:32:35.630384+0000","last_clean":"2024-09-10T20:32:35.630384+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:35.630384+0000","last_undegraded":"2024-09-10T20:32:35.630384+0000","last_fullsized":"2024-09-10T20:32:35.630384+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:05:00.172940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12434,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562408+0000","last_change":"2024-09-10T20:21:33.242850+0000","last_active":"2024-09-10T20:32:35.562408+0000","last_peered":"2024-09-10T20:32:35.562408+0000","last_clean":"2024-09-10T20:32:35.562408+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:35.562408+0000","last_undegraded":"2024-09-10T20:32:35.562408+0000","last_fullsized":"2024-09-10T20:32:35.562408+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3576","last_scrub_stamp":"2024-09-10T20:21:33.242604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:33.242604+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:03.437940+0000","scrub_duration":2026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.560571+0000","last_change":"2024-09-10T20:32:04.557049+0000","last_active":"2024-09-10T20:32:35.560571+0000","last_peered":"2024-09-10T20:32:35.560571+0000","last_clean":"2024-09-10T20:32:35.560571+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:35.560571+0000","last_undegraded":"2024-09-10T20:32:35.560571+0000","last_fullsized":"2024-09-10T20:32:35.560571+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:04.556980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:04.556980+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:25.821347+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16626,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562227+0000","last_change":"2024-09-10T20:23:18.988338+0000","last_active":"2024-09-10T20:32:35.562227+0000","last_peered":"2024-09-10T20:32:35.562227+0000","last_clean":"2024-09-10T20:32:35.562227+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:35.562227+0000","last_undegraded":"2024-09-10T20:32:35.562227+0000","last_fullsized":"2024-09-10T20:32:35.562227+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'5793","last_scrub_stamp":"2024-09-10T20:23:18.987676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:23:18.987676+0000","objects_scrubbed":5734,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:22.337098+0000","scrub_duration":5392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16270,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.564990+0000","last_change":"2024-09-10T20:32:13.667012+0000","last_active":"2024-09-10T20:32:35.564990+0000","last_peered":"2024-09-10T20:32:35.564990+0000","last_clean":"2024-09-10T20:32:35.564990+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:35.564990+0000","last_undegraded":"2024-09-10T20:32:35.564990+0000","last_fullsized":"2024-09-10T20:32:35.564990+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:13.666937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:13.666937+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.365022+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16094,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565156+0000","last_change":"2024-09-10T20:31:33.913834+0000","last_active":"2024-09-10T20:32:35.565156+0000","last_peered":"2024-09-10T20:32:35.565156+0000","last_clean":"2024-09-10T20:32:35.565156+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:35.565156+0000","last_undegraded":"2024-09-10T20:32:35.565156+0000","last_fullsized":"2024-09-10T20:32:35.565156+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:18:54.933927+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:19.359536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16130,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561475+0000","last_change":"2024-09-10T20:31:04.016822+0000","last_active":"2024-09-10T20:32:35.561475+0000","last_peered":"2024-09-10T20:32:35.561475+0000","last_clean":"2024-09-10T20:32:35.561475+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:35.561475+0000","last_undegraded":"2024-09-10T20:32:35.561475+0000","last_fullsized":"2024-09-10T20:32:35.561475+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:31:04.016716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:04.016716+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:42.225766+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12520,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562015+0000","last_change":"2024-09-10T20:31:24.089255+0000","last_active":"2024-09-10T20:32:35.562015+0000","last_peered":"2024-09-10T20:32:35.562015+0000","last_clean":"2024-09-10T20:32:35.562015+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:35.562015+0000","last_undegraded":"2024-09-10T20:32:35.562015+0000","last_fullsized":"2024-09-10T20:32:35.562015+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:31:24.089192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:24.089192+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:40.382786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16023,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565178+0000","last_change":"2024-09-10T20:21:22.722881+0000","last_active":"2024-09-10T20:32:35.565178+0000","last_peered":"2024-09-10T20:32:35.565178+0000","last_clean":"2024-09-10T20:32:35.565178+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:35.565178+0000","last_undegraded":"2024-09-10T20:32:35.565178+0000","last_fullsized":"2024-09-10T20:32:35.565178+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3447","last_scrub_stamp":"2024-09-10T20:21:22.722775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:21:22.722775+0000","objects_scrubbed":3436,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:58.675632+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.561227+0000","last_change":"2024-09-10T20:25:07.333555+0000","last_active":"2024-09-10T20:32:35.561227+0000","last_peered":"2024-09-10T20:32:35.561227+0000","last_clean":"2024-09-10T20:32:35.561227+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:35.561227+0000","last_undegraded":"2024-09-10T20:32:35.561227+0000","last_fullsized":"2024-09-10T20:32:35.561227+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7376","last_scrub_stamp":"2024-09-10T20:25:07.333245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:25:07.333245+0000","objects_scrubbed":4780,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:33:00.768557+0000","scrub_duration":4274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.565835+0000","last_change":"2024-09-10T20:32:24.954031+0000","last_active":"2024-09-10T20:32:35.565835+0000","last_peered":"2024-09-10T20:32:35.565835+0000","last_clean":"2024-09-10T20:32:35.565835+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:35.565835+0000","last_undegraded":"2024-09-10T20:32:35.565835+0000","last_fullsized":"2024-09-10T20:32:35.565835+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:24.953933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:32:24.953933+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:42.405320+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12511,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562231+0000","last_change":"2024-09-10T20:31:55.067744+0000","last_active":"2024-09-10T20:32:35.562231+0000","last_peered":"2024-09-10T20:32:35.562231+0000","last_clean":"2024-09-10T20:32:35.562231+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:35.562231+0000","last_undegraded":"2024-09-10T20:32:35.562231+0000","last_fullsized":"2024-09-10T20:32:35.562231+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:31:55.067655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:55.067655+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:21.336376+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12066,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562042+0000","last_change":"2024-09-10T20:31:44.966645+0000","last_active":"2024-09-10T20:32:35.562042+0000","last_peered":"2024-09-10T20:32:35.562042+0000","last_clean":"2024-09-10T20:32:35.562042+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:35.562042+0000","last_undegraded":"2024-09-10T20:32:35.562042+0000","last_fullsized":"2024-09-10T20:32:35.562042+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:31:44.966574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:44.966574+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:52.980990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16193,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562247+0000","last_change":"2024-09-10T20:29:35.148998+0000","last_active":"2024-09-10T20:32:35.562247+0000","last_peered":"2024-09-10T20:32:35.562247+0000","last_clean":"2024-09-10T20:32:35.562247+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:35.562247+0000","last_undegraded":"2024-09-10T20:32:35.562247+0000","last_fullsized":"2024-09-10T20:32:35.562247+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11062","last_scrub_stamp":"2024-09-10T20:29:35.148689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:29:35.148689+0000","objects_scrubbed":1044,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:19.454971+0000","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12717,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562118+0000","last_change":"2024-09-10T20:31:13.134680+0000","last_active":"2024-09-10T20:32:35.562118+0000","last_peered":"2024-09-10T20:32:35.562118+0000","last_clean":"2024-09-10T20:32:35.562118+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:35.562118+0000","last_undegraded":"2024-09-10T20:32:35.562118+0000","last_fullsized":"2024-09-10T20:32:35.562118+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:31:13.134607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:31:13.134607+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:08:40.657071+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16269,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:35.562140+0000","last_change":"2024-09-10T20:27:05.769723+0000","last_active":"2024-09-10T20:32:35.562140+0000","last_peered":"2024-09-10T20:32:35.562140+0000","last_clean":"2024-09-10T20:32:35.562140+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:35.562140+0000","last_undegraded":"2024-09-10T20:32:35.562140+0000","last_fullsized":"2024-09-10T20:32:35.562140+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'9153","last_scrub_stamp":"2024-09-10T20:27:05.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:18:54.933927+0000","last_clean_scrub_stamp":"2024-09-10T20:27:05.769371+0000","objects_scrubbed":2996,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:39.876921+0000","scrub_duration":4562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699996,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671844,"statfs":{"total":96636764160,"available":95919968256,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703652,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668188,"statfs":{"total":96636764160,"available":95916224512,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:32:45.249 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-09-10T20:18:54.933927+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=18, tm_sec=54, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-10T20:21:33.242604+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=21, tm_sec=33, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-09-10T20:32:04.556980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=4, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-10T20:23:18.987676+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=23, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-10T20:32:13.666937+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=13, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-09-10T20:18:54.933927+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=18, tm_sec=54, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-10T20:31:04.016716+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=31, tm_sec=4, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-10T20:31:24.089192+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=31, tm_sec=24, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-10T20:21:22.722775+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=21, tm_sec=22, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.250 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-10T20:25:07.333245+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=25, tm_sec=7, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.251 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-10T20:32:24.953933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=24, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.251 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-10T20:31:55.067655+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=31, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.251 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-10T20:31:44.966574+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.251 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-10T20:29:35.148689+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=29, tm_sec=35, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.251 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-09-10T20:31:13.134607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=31, tm_sec=13, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.251 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-09-10T20:27:05.769371+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=5, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=32, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:32:45.251 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-10T20:33:05.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:33:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-10T20:33:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-10T20:33:05.560 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-10T20:33:04.162591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703600,"kb_used_data":2096,"kb_used_omap":14,"kb_used_meta":2701425,"kb_avail":374783760,"statfs":{"total":386547056640,"available":383778570240,"internally_reserved":0,"allocated":2146304,"data_stored":820684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766259766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002277"},"pg_stats":[{"pgid":"1.f","version":"44'12122","reported_seq":16182,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:49.638770+0000","last_change":"2024-09-10T20:32:49.638770+0000","last_active":"2024-09-10T20:32:49.638770+0000","last_peered":"2024-09-10T20:32:49.638770+0000","last_clean":"2024-09-10T20:32:49.638770+0000","last_became_active":"2024-09-10T20:32:35.568263+0000","last_became_peered":"2024-09-10T20:32:35.568263+0000","last_unstale":"2024-09-10T20:32:49.638770+0000","last_undegraded":"2024-09-10T20:32:49.638770+0000","last_fullsized":"2024-09-10T20:32:49.638770+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12122","last_scrub_stamp":"2024-09-10T20:32:49.638662+0000","last_deep_scrub":"44'12122","last_deep_scrub_stamp":"2024-09-10T20:32:49.638662+0000","last_clean_scrub_stamp":"2024-09-10T20:32:49.638662+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:59.981834+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":48488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'12252","reported_seq":12443,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:50.871542+0000","last_change":"2024-09-10T20:32:50.871439+0000","last_active":"2024-09-10T20:32:50.871542+0000","last_peered":"2024-09-10T20:32:50.871542+0000","last_clean":"2024-09-10T20:32:50.871542+0000","last_became_active":"2024-09-10T20:21:31.068881+0000","last_became_peered":"2024-09-10T20:21:31.068881+0000","last_unstale":"2024-09-10T20:32:50.871542+0000","last_undegraded":"2024-09-10T20:32:50.871542+0000","last_fullsized":"2024-09-10T20:32:50.871542+0000","mapping_epoch":35,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12252","last_scrub_stamp":"2024-09-10T20:32:50.871326+0000","last_deep_scrub":"44'12252","last_deep_scrub_stamp":"2024-09-10T20:32:50.871326+0000","last_clean_scrub_stamp":"2024-09-10T20:32:50.871326+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:39:49.561742+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'12140","reported_seq":16238,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:46.363040+0000","last_change":"2024-09-10T20:32:46.363040+0000","last_active":"2024-09-10T20:32:46.363040+0000","last_peered":"2024-09-10T20:32:46.363040+0000","last_clean":"2024-09-10T20:32:46.363040+0000","last_became_active":"2024-09-10T20:32:03.458769+0000","last_became_peered":"2024-09-10T20:32:03.458769+0000","last_unstale":"2024-09-10T20:32:46.363040+0000","last_undegraded":"2024-09-10T20:32:46.363040+0000","last_fullsized":"2024-09-10T20:32:46.363040+0000","mapping_epoch":57,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12140","last_scrub_stamp":"2024-09-10T20:32:46.362933+0000","last_deep_scrub":"44'12140","last_deep_scrub_stamp":"2024-09-10T20:32:46.362933+0000","last_clean_scrub_stamp":"2024-09-10T20:32:46.362933+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:04.642041+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"44'12406","reported_seq":16634,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:45.880242+0000","last_change":"2024-09-10T20:32:45.880242+0000","last_active":"2024-09-10T20:32:45.880242+0000","last_peered":"2024-09-10T20:32:45.880242+0000","last_clean":"2024-09-10T20:32:45.880242+0000","last_became_active":"2024-09-10T20:23:13.161057+0000","last_became_peered":"2024-09-10T20:23:13.161057+0000","last_unstale":"2024-09-10T20:32:45.880242+0000","last_undegraded":"2024-09-10T20:32:45.880242+0000","last_fullsized":"2024-09-10T20:32:45.880242+0000","mapping_epoch":37,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12406","last_scrub_stamp":"2024-09-10T20:32:45.880155+0000","last_deep_scrub":"44'12406","last_deep_scrub_stamp":"2024-09-10T20:32:45.880155+0000","last_clean_scrub_stamp":"2024-09-10T20:32:45.880155+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:50.018353+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"44'12167","reported_seq":16278,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:45.597530+0000","last_change":"2024-09-10T20:32:45.597530+0000","last_active":"2024-09-10T20:32:45.597530+0000","last_peered":"2024-09-10T20:32:45.597530+0000","last_clean":"2024-09-10T20:32:45.597530+0000","last_became_active":"2024-09-10T20:32:13.494344+0000","last_became_peered":"2024-09-10T20:32:13.494344+0000","last_unstale":"2024-09-10T20:32:45.597530+0000","last_undegraded":"2024-09-10T20:32:45.597530+0000","last_fullsized":"2024-09-10T20:32:45.597530+0000","mapping_epoch":59,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12167","last_scrub_stamp":"2024-09-10T20:32:45.597437+0000","last_deep_scrub":"44'12167","last_deep_scrub_stamp":"2024-09-10T20:32:45.597437+0000","last_clean_scrub_stamp":"2024-09-10T20:32:45.597437+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:02:06.447920+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"44'12098","reported_seq":16102,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:48.624949+0000","last_change":"2024-09-10T20:32:48.624949+0000","last_active":"2024-09-10T20:32:48.624949+0000","last_peered":"2024-09-10T20:32:48.624949+0000","last_clean":"2024-09-10T20:32:48.624949+0000","last_became_active":"2024-09-10T20:31:33.853579+0000","last_became_peered":"2024-09-10T20:31:33.853579+0000","last_unstale":"2024-09-10T20:32:48.624949+0000","last_undegraded":"2024-09-10T20:32:48.624949+0000","last_fullsized":"2024-09-10T20:32:48.624949+0000","mapping_epoch":51,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12098","last_scrub_stamp":"2024-09-10T20:32:48.624824+0000","last_deep_scrub":"44'12098","last_deep_scrub_stamp":"2024-09-10T20:32:48.624824+0000","last_clean_scrub_stamp":"2024-09-10T20:32:48.624824+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:00.846810+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":48392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"44'12068","reported_seq":16138,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:44.100624+0000","last_change":"2024-09-10T20:32:44.100624+0000","last_active":"2024-09-10T20:32:44.100624+0000","last_peered":"2024-09-10T20:32:44.100624+0000","last_clean":"2024-09-10T20:32:44.100624+0000","last_became_active":"2024-09-10T20:31:03.219400+0000","last_became_peered":"2024-09-10T20:31:03.219400+0000","last_unstale":"2024-09-10T20:32:44.100624+0000","last_undegraded":"2024-09-10T20:32:44.100624+0000","last_fullsized":"2024-09-10T20:32:44.100624+0000","mapping_epoch":45,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12068","last_scrub_stamp":"2024-09-10T20:32:44.100529+0000","last_deep_scrub":"44'12068","last_deep_scrub_stamp":"2024-09-10T20:32:44.100529+0000","last_clean_scrub_stamp":"2024-09-10T20:32:44.100529+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:31.529431+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'12291","reported_seq":12528,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:47.876042+0000","last_change":"2024-09-10T20:32:47.876042+0000","last_active":"2024-09-10T20:32:47.876042+0000","last_peered":"2024-09-10T20:32:47.876042+0000","last_clean":"2024-09-10T20:32:47.876042+0000","last_became_active":"2024-09-10T20:31:23.808771+0000","last_became_peered":"2024-09-10T20:31:23.808771+0000","last_unstale":"2024-09-10T20:32:47.876042+0000","last_undegraded":"2024-09-10T20:32:47.876042+0000","last_fullsized":"2024-09-10T20:32:47.876042+0000","mapping_epoch":49,"log_start":"32'2200","ondisk_log_start":"32'2200","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12291","last_scrub_stamp":"2024-09-10T20:32:47.875936+0000","last_deep_scrub":"44'12291","last_deep_scrub_stamp":"2024-09-10T20:32:47.875936+0000","last_clean_scrub_stamp":"2024-09-10T20:32:47.875936+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:05:13.832168+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"44'12000","reported_seq":16031,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:46.584565+0000","last_change":"2024-09-10T20:32:46.584565+0000","last_active":"2024-09-10T20:32:46.584565+0000","last_peered":"2024-09-10T20:32:46.584565+0000","last_clean":"2024-09-10T20:32:46.584565+0000","last_became_active":"2024-09-10T20:21:22.161048+0000","last_became_peered":"2024-09-10T20:21:22.161048+0000","last_unstale":"2024-09-10T20:32:46.584565+0000","last_undegraded":"2024-09-10T20:32:46.584565+0000","last_fullsized":"2024-09-10T20:32:46.584565+0000","mapping_epoch":33,"log_start":"29'1900","ondisk_log_start":"29'1900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12000","last_scrub_stamp":"2024-09-10T20:32:46.584468+0000","last_deep_scrub":"44'12000","last_deep_scrub_stamp":"2024-09-10T20:32:46.584468+0000","last_clean_scrub_stamp":"2024-09-10T20:32:46.584468+0000","objects_scrubbed":1,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:16.983078+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'12142","reported_seq":16171,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:45.367180+0000","last_change":"2024-09-10T20:32:45.367180+0000","last_active":"2024-09-10T20:32:45.367180+0000","last_peered":"2024-09-10T20:32:45.367180+0000","last_clean":"2024-09-10T20:32:45.367180+0000","last_became_active":"2024-09-10T20:25:02.385224+0000","last_became_peered":"2024-09-10T20:25:02.385224+0000","last_unstale":"2024-09-10T20:32:45.367180+0000","last_undegraded":"2024-09-10T20:32:45.367180+0000","last_fullsized":"2024-09-10T20:32:45.367180+0000","mapping_epoch":39,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12142","last_scrub_stamp":"2024-09-10T20:32:45.367084+0000","last_deep_scrub":"44'12142","last_deep_scrub_stamp":"2024-09-10T20:32:45.367084+0000","last_clean_scrub_stamp":"2024-09-10T20:32:45.367084+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:41:08.710401+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"44'12110","reported_seq":16178,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:47.620362+0000","last_change":"2024-09-10T20:32:47.620362+0000","last_active":"2024-09-10T20:32:47.620362+0000","last_peered":"2024-09-10T20:32:47.620362+0000","last_clean":"2024-09-10T20:32:47.620362+0000","last_became_active":"2024-09-10T20:32:23.537316+0000","last_became_peered":"2024-09-10T20:32:23.537316+0000","last_unstale":"2024-09-10T20:32:47.620362+0000","last_undegraded":"2024-09-10T20:32:47.620362+0000","last_fullsized":"2024-09-10T20:32:47.620362+0000","mapping_epoch":61,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12110","last_scrub_stamp":"2024-09-10T20:32:47.620222+0000","last_deep_scrub":"44'12110","last_deep_scrub_stamp":"2024-09-10T20:32:47.620222+0000","last_clean_scrub_stamp":"2024-09-10T20:32:47.620222+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:46:53.404579+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"44'12164","reported_seq":12519,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:49.880092+0000","last_change":"2024-09-10T20:32:49.880092+0000","last_active":"2024-09-10T20:32:49.880092+0000","last_peered":"2024-09-10T20:32:49.880092+0000","last_clean":"2024-09-10T20:32:49.880092+0000","last_became_active":"2024-09-10T20:31:54.434320+0000","last_became_peered":"2024-09-10T20:31:54.434320+0000","last_unstale":"2024-09-10T20:32:49.880092+0000","last_undegraded":"2024-09-10T20:32:49.880092+0000","last_fullsized":"2024-09-10T20:32:49.880092+0000","mapping_epoch":55,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12164","last_scrub_stamp":"2024-09-10T20:32:49.879986+0000","last_deep_scrub":"44'12164","last_deep_scrub_stamp":"2024-09-10T20:32:49.879986+0000","last_clean_scrub_stamp":"2024-09-10T20:32:49.879986+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:03:08.477675+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"44'11830","reported_seq":12074,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:44.922760+0000","last_change":"2024-09-10T20:32:44.922760+0000","last_active":"2024-09-10T20:32:44.922760+0000","last_peered":"2024-09-10T20:32:44.922760+0000","last_clean":"2024-09-10T20:32:44.922760+0000","last_became_active":"2024-09-10T20:31:43.886128+0000","last_became_peered":"2024-09-10T20:31:43.886128+0000","last_unstale":"2024-09-10T20:32:44.922760+0000","last_undegraded":"2024-09-10T20:32:44.922760+0000","last_fullsized":"2024-09-10T20:32:44.922760+0000","mapping_epoch":53,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'11830","last_scrub_stamp":"2024-09-10T20:32:44.922670+0000","last_deep_scrub":"44'11830","last_deep_scrub_stamp":"2024-09-10T20:32:44.922670+0000","last_clean_scrub_stamp":"2024-09-10T20:32:44.922670+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:02:32.754882+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'12092","reported_seq":16201,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:46.833514+0000","last_change":"2024-09-10T20:32:46.833514+0000","last_active":"2024-09-10T20:32:46.833514+0000","last_peered":"2024-09-10T20:32:46.833514+0000","last_clean":"2024-09-10T20:32:46.833514+0000","last_became_active":"2024-09-10T20:29:32.300730+0000","last_became_peered":"2024-09-10T20:29:32.300730+0000","last_unstale":"2024-09-10T20:32:46.833514+0000","last_undegraded":"2024-09-10T20:32:46.833514+0000","last_fullsized":"2024-09-10T20:32:46.833514+0000","mapping_epoch":43,"log_start":"32'2000","ondisk_log_start":"32'2000","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12092","last_scrub_stamp":"2024-09-10T20:32:46.833411+0000","last_deep_scrub":"44'12092","last_deep_scrub_stamp":"2024-09-10T20:32:46.833411+0000","last_clean_scrub_stamp":"2024-09-10T20:32:46.833411+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:07:13.804493+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'12447","reported_seq":12725,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:48.831297+0000","last_change":"2024-09-10T20:32:48.831297+0000","last_active":"2024-09-10T20:32:48.831297+0000","last_peered":"2024-09-10T20:32:48.831297+0000","last_clean":"2024-09-10T20:32:48.831297+0000","last_became_active":"2024-09-10T20:31:12.736862+0000","last_became_peered":"2024-09-10T20:31:12.736862+0000","last_unstale":"2024-09-10T20:32:48.831297+0000","last_undegraded":"2024-09-10T20:32:48.831297+0000","last_fullsized":"2024-09-10T20:32:48.831297+0000","mapping_epoch":47,"log_start":"32'2400","ondisk_log_start":"32'2400","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12447","last_scrub_stamp":"2024-09-10T20:32:48.831187+0000","last_deep_scrub":"44'12447","last_deep_scrub_stamp":"2024-09-10T20:32:48.831187+0000","last_clean_scrub_stamp":"2024-09-10T20:32:48.831187+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:44.094799+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"44'12123","reported_seq":16277,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-10T20:32:51.917553+0000","last_change":"2024-09-10T20:32:51.917553+0000","last_active":"2024-09-10T20:32:51.917553+0000","last_peered":"2024-09-10T20:32:51.917553+0000","last_clean":"2024-09-10T20:32:51.917553+0000","last_became_active":"2024-09-10T20:27:00.553749+0000","last_became_peered":"2024-09-10T20:27:00.553749+0000","last_unstale":"2024-09-10T20:32:51.917553+0000","last_undegraded":"2024-09-10T20:32:51.917553+0000","last_fullsized":"2024-09-10T20:32:51.917553+0000","mapping_epoch":41,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'12123","last_scrub_stamp":"2024-09-10T20:32:51.917403+0000","last_deep_scrub":"44'12123","last_deep_scrub_stamp":"2024-09-10T20:32:51.917403+0000","last_clean_scrub_stamp":"2024-09-10T20:32:51.917403+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:46:36.040240+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56952,"num_write_kb":141640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117160,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699996,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93671844,"statfs":{"total":96636764160,"available":95919968256,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703644,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93668196,"statfs":{"total":96636764160,"available":95916232704,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595868,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775972,"statfs":{"total":96636764160,"available":96026595328,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704092,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93667748,"statfs":{"total":96636764160,"available":95915773952,"internally_reserved":0,"allocated":536576,"data_stored":205171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":720432847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:33:05.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-10T20:33:05.864 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-10T20:33:05.865 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-10T20:33:05.865 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-10T20:33:05.865 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:11.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-10T20:33:11.967 INFO:tasks.ceph.osd.0:Stopped 2024-09-10T20:33:11.968 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-10T20:33:11.968 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:12.051 INFO:tasks.ceph.osd.1:Stopped 2024-09-10T20:33:12.052 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-10T20:33:12.052 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:16.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-10T20:33:16.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-10T20:33:18.153 INFO:tasks.ceph.osd.2:Stopped 2024-09-10T20:33:18.154 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-10T20:33:18.154 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:22.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-10T20:33:22.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-10T20:33:24.255 INFO:tasks.ceph.osd.3:Stopped 2024-09-10T20:33:24.255 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-10T20:33:24.256 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-10T20:33:24.256 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:24.290 INFO:tasks.ceph.mgr.x:Stopped 2024-09-10T20:33:24.290 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-10T20:33:24.290 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T20:33:24.290 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:24.345 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T20:33:24.345 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T20:33:24.345 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:24.398 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T20:33:24.398 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T20:33:24.398 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:33:24.452 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T20:33:24.452 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-10T20:33:24.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-10T20:33:24.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:33:24.731 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-10T20:33:25.335 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:33:25.335 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-10T20:33:25.434 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:33:25.434 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-10T20:33:25.513 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:33:25.513 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-10T20:33:25.586 INFO:tasks.ceph:Archiving mon data... 2024-09-10T20:33:25.588 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553/data/mon.a.tgz 2024-09-10T20:33:25.588 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-10T20:33:25.608 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.1BMIwFQJwf 2024-09-10T20:33:25.608 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.1BMIwFQJwf 2024-09-10T20:33:25.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.1BMIwFQJwf 2024-09-10T20:33:25.877 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.1BMIwFQJwf is 347KB 2024-09-10T20:33:25.977 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.1BMIwFQJwf 2024-09-10T20:33:25.992 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553/data/mon.b.tgz 2024-09-10T20:33:25.993 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-10T20:33:26.046 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.gZZMmzerhc 2024-09-10T20:33:26.047 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.gZZMmzerhc 2024-09-10T20:33:26.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.gZZMmzerhc 2024-09-10T20:33:26.256 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.gZZMmzerhc is 360KB 2024-09-10T20:33:26.351 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.gZZMmzerhc 2024-09-10T20:33:26.366 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553/data/mon.c.tgz 2024-09-10T20:33:26.367 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-10T20:33:26.422 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.iPxuD7ugeh 2024-09-10T20:33:26.422 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.iPxuD7ugeh 2024-09-10T20:33:26.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.iPxuD7ugeh 2024-09-10T20:33:26.635 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.iPxuD7ugeh is 360KB 2024-09-10T20:33:26.702 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.iPxuD7ugeh 2024-09-10T20:33:26.717 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-10T20:33:26.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T20:33:26.822 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T20:33:26.822 INFO:tasks.ceph:Archiving crash dumps... 2024-09-10T20:33:26.824 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553/remote/smithi103/crash 2024-09-10T20:33:26.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T20:33:26.886 INFO:tasks.ceph:Compressing logs... 2024-09-10T20:33:26.886 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T20:33:26.965 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-10T20:33:26.966 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68892.log 2024-09-10T20:33:26.966 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54058.log 2024-09-10T20:33:27.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-10T20:33:27.217 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.68892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39740.log 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68892.log.gz 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61245.log 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54058.log.gz 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59142.log 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39740.log.gz 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56032.log 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61245.log.gz 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38298.log 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59142.log.gz 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60867.log 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56032.log.gz 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53428.log 2024-09-10T20:33:27.218 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38298.log.gz 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39838.log 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60867.log.gz 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53596.log 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53428.log.gz 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58848.log 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39838.log.gz 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55612.log 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53596.log.gz 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39333.log 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58848.log.gz 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44776.log 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55612.log.gz 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51370.log 2024-09-10T20:33:27.219 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39333.log.gz 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60234.log 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44776.log.gz 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61329.log 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51370.log.gz 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52798.log 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60234.log.gz 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61329.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61329.log.gz /var/log/ceph/ceph-client.admin.68346.log 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56830.log 2024-09-10T20:33:27.220 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52798.log.gz 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54268.log 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68346.log.gz 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68556.log 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56830.log.gz 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50530.log 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54268.log.gz 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52672.log 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68556.log.gz 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63009.log 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50530.log.gz 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57712.log 2024-09-10T20:33:27.221 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52672.log.gz 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65994.log 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63009.log.gz 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40717.log 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57712.log.gz 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62379.log 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65994.log.gz 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65406.log 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40717.log.gz 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62379.log.gz 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63977.log 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65406.log.gz 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52714.log 2024-09-10T20:33:27.222 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37808.log 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63977.log.gz 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49270.log 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52714.log.gz 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69438.log 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37808.log.gz 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45723.log 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49270.log.gz 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55780.log 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69438.log.gz 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59688.log 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45723.log.gz 2024-09-10T20:33:27.223 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69186.log 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55780.log.gz 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45359.log 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59688.log.gz 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61161.log 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69186.log.gz 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48976.log 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45359.log.gz 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67464.log 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61161.log.gz 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37640.log 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48976.log.gz 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65279.log 2024-09-10T20:33:27.224 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67464.log.gz 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70111.log 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37640.log.gz 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33524.log 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65279.log.gz 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70111.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.70111.log.gz 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51328.log 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33524.log.gz 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67674.log 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56410.log 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51328.log.gz 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57586.log 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67674.log.gz 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65195.log 2024-09-10T20:33:27.225 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56410.log.gz 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58932.log 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57586.log.gz 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65195.log.gz 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60570.log 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48598.log 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58932.log.gz 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54478.log 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60570.log.gz 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48598.log.gz 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71582.log 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53050.log 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54478.log.gz 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57334.log 2024-09-10T20:33:27.226 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71582.log.gz 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57166.log 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53050.log.gz 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54352.log 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57334.log.gz 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43684.log 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57166.log.gz 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54352.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.54352.log.gz /var/log/ceph/ceph-client.admin.70910.log 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43684.log.gz 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55360.log 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70910.log.gz 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67548.log 2024-09-10T20:33:27.227 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61875.log 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55360.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67548.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59100.log 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66288.log 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61875.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43432.log 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59100.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45028.log 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66288.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60192.log 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43432.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62421.log 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45028.log.gz 2024-09-10T20:33:27.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46683.log 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60192.log.gz 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58258.log 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62421.log.gz 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60486.log 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46683.log.gz 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58258.log.gz 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60108.log 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38889.log 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60486.log.gz 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38321.log 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60108.log.gz 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50236.log 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38889.log.gz 2024-09-10T20:33:27.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53848.log 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38321.log.gz 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64271.log 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50236.log.gz 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53848.log.gz 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52252.log 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64271.log.gz 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44188.log 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52252.log.gz 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68598.log 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gz 2024-09-10T20:33:27.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68808.log 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44188.log.gz 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61623.log 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68598.log.gz 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43055.log 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68808.log.gz 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46535.log 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61623.log.gz 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69816.log 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43055.log.gz 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46794.log 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46535.log.gz 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49144.log 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69816.log.gz 2024-09-10T20:33:27.231 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46794.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46794.log.gz 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63682.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49144.log.gz 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53890.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63598.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63682.log.gz 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53890.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27380.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53890.log.gz 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40643.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63598.log.gz 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.tmp-client.admin.27380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43897.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27380.log.gz 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59772.log 2024-09-10T20:33:27.232 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40643.log.gz 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70826.log 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43897.log.gz 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47842.log 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59772.log.gz 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63893.log 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70826.log.gz 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.47842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43180.log 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47842.log.gz 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61665.log 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63893.log.gz 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63766.log 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43180.log.gz 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50614.log 2024-09-10T20:33:27.233 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61665.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70952.log 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63766.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63093.log 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50614.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39629.log 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70952.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63093.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45488.log 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39629.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38255.log: -- /var/log/ceph/ceph-client.admin.44902.log 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54898.log 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45488.log.gz 2024-09-10T20:33:27.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65153.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44902.log.gz 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56116.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54898.log.gz 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69060.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65153.log.gz 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69901.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56116.log.gz 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44230.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69060.log.gz 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55318.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69901.log.gz 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64985.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44230.log.gz 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44986.log 2024-09-10T20:33:27.235 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55318.log.gz 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53932.log 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64985.log.gz 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59856.log 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44986.log.gz 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46572.log 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53932.log.gz 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64859.log 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59856.log.gz 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61497.log 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46572.log.gz 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64859.log.gz 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63851.log 2024-09-10T20:33:27.236 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61497.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60276.log 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48136.log 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63851.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50572.log 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60276.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60024.log 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48136.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45602.log 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50572.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52966.log 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60024.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58764.log 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45602.log.gz 2024-09-10T20:33:27.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60360.log 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52966.log.gz 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58764.log.gz 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65448.log 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60360.log.gz 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70616.log 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66792.log 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65448.log.gz 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67380.log 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70616.log.gz 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59016.log 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66792.log.gz 2024-09-10T20:33:27.238 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59982.log 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67380.log.gz 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66582.log 0.0% 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59016.log.gz 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64481.log 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59982.log.gz 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59478.log 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66582.log.gz 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62001.log 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64481.log.gz 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54016.log 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59478.log.gz 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55990.log 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62001.log.gz 2024-09-10T20:33:27.239 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59436.log 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54016.log.gz 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61455.log 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55990.log.gz 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57502.log 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59436.log.gz 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53134.log 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61455.log.gz 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68976.log 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57502.log.gz 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63346.log 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53134.log.gz 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59604.log 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68976.log.gz 2024-09-10T20:33:27.240 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66918.log 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63346.log.gz 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43433.log 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59604.log.gz 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54184.log 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66918.log.gz 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43433.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.48178.log -- replaced with /var/log/ceph/ceph-client.admin.43433.log.gz 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71162.log 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54184.log.gz 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58552.log 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48178.log.gz 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38396.log 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71162.log.gz 2024-09-10T20:33:27.241 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62631.log 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58552.log.gz 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68094.log 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38396.log.gz 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45280.log 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62631.log.gz 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39666.log 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68094.log.gz 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45280.log.gz 2024-09-10T20:33:27.242 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39666.log.gz 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38481.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53722.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38755.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38481.log.gz 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50740.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53722.log.gz 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70195.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38755.log.gz 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50740.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50740.log.gz -5 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66498.log 2024-09-10T20:33:27.243 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70195.log.gz 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69606.log 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66498.log.gz 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47926.log 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69606.log.gz 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48892.log 2024-09-10T20:33:27.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.47926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44566.log 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47926.log.gz 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56872.log 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48892.log.gz 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44566.log.gz 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51622.log 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71330.log 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56872.log.gz 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56158.log 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51622.log.gz 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64187.log 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71330.log.gz 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48640.log 2024-09-10T20:33:27.245 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56158.log.gz 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46387.log 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64187.log.gz 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55276.log 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48640.log.gz 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58384.log 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46387.log.gz 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57124.log 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55276.log.gz 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64775.log 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58384.log.gz 2024-09-10T20:33:27.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42847.log 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57124.log.gz 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44734.log 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64775.log.gz 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.42847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47884.log 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42847.log.gz 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45154.log 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44734.log.gz 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.47884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66960.log 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47884.log.gz 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70700.log 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45154.log.gz 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64103.log 2024-09-10T20:33:27.247 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66960.log.gz 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55738.log 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70700.log.gz 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64103.log.gz 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43978.log 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55738.log.gz 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61203.log 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57040.log 2024-09-10T20:33:27.248 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43978.log.gz 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67716.log 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61203.log.gz 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46014.log 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57040.log.gz 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46868.log 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67716.log.gz 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58890.log 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46014.log.gz 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43557.log 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46868.log.gz 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47800.log 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58890.log.gz 2024-09-10T20:33:27.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59184.log 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43557.log.gz 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.47800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56368.log 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47800.log.gz 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49690.log 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59184.log.gz 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55066.log 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56368.log.gz 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54562.log 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49690.log.gz 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55066.log.gz 2024-09-10T20:33:27.250 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70153.log 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54562.log.gz 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58300.log 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70658.log 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70153.log.gz 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71246.log 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58300.log.gz 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60825.log 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70658.log.gz 2024-09-10T20:33:27.251 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70742.log 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71246.log.gz 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58090.log 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60825.log.gz 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59268.log 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70742.log.gz 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44524.log 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58090.log.gz 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43054.log 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59268.log.gz 2024-09-10T20:33:27.252 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45358.log 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44524.log.gz 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60318.log 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43054.log.gz 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57208.log 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45358.log.gz 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38169.log 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60318.log.gz 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.57208.log: /var/log/ceph/ceph-client.admin.46757.log 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57208.log.gz 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48052.log 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38169.log.gz 2024-09-10T20:33:27.253 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52336.log 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46757.log.gz 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64649.log 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48052.log.gz 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53512.log 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52336.log.gz 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53764.log 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64649.log.gz 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70322.log 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53512.log.gz 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52840.log 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53764.log.gz 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69354.log 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70322.log.gz 2024-09-10T20:33:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57880.log 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52840.log.gz 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62967.log 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69354.log.gz 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37471.log 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57880.log.gz 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33404.log 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62967.log.gz 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40310.log 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37471.log.gz 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33404.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.33404.log.gz --verbose 2024-09-10T20:33:27.255 INFO:teuthology.orchestra.run.smithi103.stderr: -- /var/log/ceph/ceph-client.admin.37768.log 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65069.log 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40310.log.gz 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64397.log 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37768.log.gz 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66750.log 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65069.log.gz 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65658.log 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64397.log.gz 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39148.log 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66750.log.gz 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37245.log 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65658.log.gz 2024-09-10T20:33:27.256 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.39148.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39148.log.gz 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62589.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37245.log.gz 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46275.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62589.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62589.log.gz 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63135.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46275.log.gz 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52546.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33267.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63135.log.gz 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67968.log 2024-09-10T20:33:27.257 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52546.log.gz 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65910.log 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33267.log.gz 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68514.log 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67968.log.gz 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39074.log 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65910.log.gz 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38127.log 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68514.log.gz 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52420.log 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39074.log.gz 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45898.log 2024-09-10T20:33:27.258 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38127.log.gz 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/ceph-client.admin.52420.log: --verbose -- /var/log/ceph/ceph-client.admin.38565.log 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52420.log.gz 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46201.log 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45898.log.gz 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68724.log 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38565.log.gz 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52294.log 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46201.log.gz 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46461.log 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68724.log.gz 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59352.log 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52294.log.gz 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54226.log 2024-09-10T20:33:27.259 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46461.log.gz 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38713.log 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59352.log.gz 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49648.log 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54226.log.gz 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53554.log 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38713.log.gz 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.49648.log: -- /var/log/ceph/ceph-client.admin.48262.log 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49648.log.gz 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65742.log 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53554.log.gz 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48262.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.48262.log.gz -- 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/ceph-client.admin.37809.log 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57544.log 2024-09-10T20:33:27.260 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65742.log.gz 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43685.log 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37809.log.gz 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48388.log 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57544.log.gz 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53218.log 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43685.log.gz 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52168.log 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48388.log.gz 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49522.log 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53218.log.gz 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45872.log 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52168.log.gz 2024-09-10T20:33:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69144.log 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49522.log.gz 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63724.log 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45872.log.gz 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63388.log 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69144.log.gz 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52210.log 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63724.log.gz 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61581.log 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63388.log.gz 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66876.log 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52210.log.gz 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43053.log 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61581.log.gz 2024-09-10T20:33:27.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69732.log 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66876.log.gz 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43053.log.gz 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48430.log 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49186.log 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69732.log.gz 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48430.log.gz 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63808.log 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53008.log 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49186.log.gz 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63808.log.gz 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57460.log 2024-09-10T20:33:27.263 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53008.log.gz 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65490.log 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37081.log 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57460.log.gz 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52000.log 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65490.log.gz 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49438.log 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37081.log.gz 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58974.log 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52000.log.gz 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37959.log 2024-09-10T20:33:27.264 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49438.log.gz 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39703.log 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58974.log.gz 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58174.log 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37959.log.gz 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49816.log 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39703.log.gz 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61119.log 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58174.log.gz 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68388.log 2024-09-10T20:33:27.265 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49816.log.gz 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54940.log 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61119.log.gz 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52084.log 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68388.log.gz 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61413.log 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54940.log.gz 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67254.log 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52084.log.gz 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67800.log 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61413.log.gz 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70574.log 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67254.log.gz 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38001.log 2024-09-10T20:33:27.266 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67800.log.gz 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48472.log 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70574.log.gz 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40162.log 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38001.log.gz 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49354.log 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48472.log.gz 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58468.log 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40162.log.gz 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58426.log 2024-09-10T20:33:27.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49354.log.gz 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59562.log 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58468.log.gz 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60951.log 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58426.log.gz 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71414.log 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59562.log.gz 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61749.log 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60951.log.gz 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65826.log 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71414.log.gz 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49018.log 2024-09-10T20:33:27.268 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61749.log.gz 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43141.log 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65826.log.gz 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37472.log 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49018.log.gz 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56914.log 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43141.log.gz 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39000.log 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37472.log.gz 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57376.log 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56914.log.gz 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71036.log 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39000.log.gz 2024-09-10T20:33:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61035.log 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57376.log.gz 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65532.log 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71036.log.gz 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48220.log 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61035.log.gz 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45489.log 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65532.log.gz 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66834.log 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48220.log.gz 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54520.log 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45489.log.gz 2024-09-10T20:33:27.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38629.log 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66834.log.gz 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58510.log 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54520.log.gz 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48346.log 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38629.log.gz 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70406.log 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58510.log.gz 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63514.log 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48346.log.gz 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46312.log 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70406.log.gz 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37641.log 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63514.log.gz 2024-09-10T20:33:27.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53680.log 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46312.log.gz 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70994.log 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37641.log.gz 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55234.log 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53680.log.gz 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69858.log 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70994.log.gz 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71078.log 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55234.log.gz 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54772.log 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69858.log.gz 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71078.log.gz 2024-09-10T20:33:27.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67128.log 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54772.log.gz 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60066.log 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67128.log.gz 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46831.log 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60066.log.gz 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43811.log 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42927.log 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46831.log.gz 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2024-09-10T20:33:27.273 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43811.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.42927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39880.log 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42927.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51832.log 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67086.log 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39880.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54856.log 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51832.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr: 95.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39796.log 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67086.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64061.log 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54856.log.gz 2024-09-10T20:33:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44020.log 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39796.log.gz 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44272.log 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64061.log.gz 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44020.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44020.log.gz 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56788.log 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44272.log.gz 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60528.log 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56788.log.gz 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56284.log 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60528.log.gz 2024-09-10T20:33:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51076.log 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46350.log 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56284.log.gz 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51748.log 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51076.log.gz 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53974.log 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46350.log.gz 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51748.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.49942.log -- replaced with /var/log/ceph/ceph-client.admin.51748.log.gz 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59310.log 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53974.log.gz 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61707.log 2024-09-10T20:33:27.276 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49942.log.gz 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68640.log 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59310.log.gz 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70448.log 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61707.log.gz 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54394.log 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68640.log.gz 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70448.log.gz 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66246.log 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46090.log 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54394.log.gz 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53470.log 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66246.log.gz 2024-09-10T20:33:27.277 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.46090.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65027.log 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46090.log.gz 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49858.log 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53470.log.gz 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65027.log.gzgzip 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69985.log 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49858.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49858.log.gz 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51244.log 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69985.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.69985.log.gz 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/ceph-client.admin.56074.log 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37917.log 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51244.log.gz 2024-09-10T20:33:27.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49900.log 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56074.log.gz 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58594.log 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37917.log.gz 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49900.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.49900.log.gz 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/ceph-client.admin.37470.log 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39903.log 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58594.log.gz 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70784.log 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37470.log.gz 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61371.log 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39903.log.gz 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2024-09-10T20:33:27.279 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70784.log.gz 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49396.log/var/log/ceph/ceph-client.admin.61371.log: 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61371.log.gz 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54604.log 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69690.log 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49396.log.gz 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44314.log 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54604.log.gz 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.69690.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.61959.log 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69690.log.gz 2024-09-10T20:33:27.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51160.log 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44314.log.gz 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71372.log 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61959.log.gz 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45238.log 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51160.log.gz 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71372.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.33484.log 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71372.log.gz 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40828.log 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45238.log.gz 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66624.log 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33484.log.gz 2024-09-10T20:33:27.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61917.log 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40828.log.gz 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60150.log 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66624.log.gz 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51496.log 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61917.log.gz 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60150.log.gz 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62505.log 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68052.log 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51496.log.gz 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66666.log 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62505.log.gz 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56620.log 2024-09-10T20:33:27.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68052.log.gz 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57292.log 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66666.log.gz 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37329.log 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56620.log.gz 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55570.log 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57292.log.gz 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47968.log 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37329.log.gz 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60612.log 2024-09-10T20:33:27.283 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55570.log.gz 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.47968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55948.log 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47968.log.gz 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45617.log 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60612.log.gz 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55948.log.gz 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55402.log 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66372.log 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45617.log.gz 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45940.log 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55402.log.gz 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58006.log 2024-09-10T20:33:27.284 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66372.log.gz 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50908.log 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45940.log.gz 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58006.log.gz 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64607.log 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50908.log.gz 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59814.log 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64607.log.gz 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70069.log 2024-09-10T20:33:27.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68178.log 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59814.log.gz 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39555.log 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70069.log.gz 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39037.log 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68178.log.gz 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53092.log 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39555.log.gz 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61791.log 2024-09-10T20:33:27.286 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39037.log.gz 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67926.log 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53092.log.gz 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61791.log.gz 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41901.log/var/log/ceph/ceph-client.admin.67926.log: 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67926.log.gz 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71622.log 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.41901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58680.log 2024-09-10T20:33:27.287 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41901.log.gz 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60993.log 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71622.log.gz 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62253.log 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58680.log.gz 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64733.log 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60993.log.gz 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57082.log/var/log/ceph/ceph-client.admin.62253.log: 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62253.log.gz 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70027.log 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64733.log.gz 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54982.log 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57082.log.gz 2024-09-10T20:33:27.288 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.70027.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62673.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70027.log.gz 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49774.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54982.log.gz 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62673.log.gz 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38926.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71540.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49774.log.gz 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38926.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38926.log.gz 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61077.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71540.log.gz 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46238.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70490.log 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61077.log.gz 2024-09-10T20:33:27.289 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66204.log 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46238.log.gz 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52756.log 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70490.log.gz 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66204.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.66204.log.gz -5 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60402.log 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55444.log 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52756.log.gz 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43309.log 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60402.log.gz 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59058.log 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55444.log.gz 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50824.log 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43309.log.gz 2024-09-10T20:33:27.290 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59058.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.59058.log.gz 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63640.log 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45594.log 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50824.log.gz 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43558.log 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63640.log.gz 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45463.log 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45594.log.gz 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43558.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43558.log.gz -5 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48010.log 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45463.log.gz 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53386.log 2024-09-10T20:33:27.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46164.log 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48010.log.gz 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37639.log 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53386.log.gz 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61539.log 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37639.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37639.log.gz -5 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57796.log 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67170.log 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61539.log.gz 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51412.log 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57796.log.gz 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67170.log.gz 2024-09-10T20:33:27.292 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50194.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44146.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51412.log.gz 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57670.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50194.log.gz 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39259.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44146.log.gz 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68220.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57670.log.gz 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63219.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39259.log.gz 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66120.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68220.log.gz 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50152.log 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63219.log.gz 2024-09-10T20:33:27.293 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62169.log 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66120.log.gz 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52126.log 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50152.log.gz 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63556.log 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62169.log.gz 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52126.log.gzgzip 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59226.log 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58048.log 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63556.log.gz 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71498.log 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59226.log.gz 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51874.log 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58048.log.gz 2024-09-10T20:33:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65952.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71498.log.gz 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54730.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51874.log.gz 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40273.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65952.log.gz 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40569.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54730.log.gz 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40273.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40273.log.gz 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39977.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40569.log.gz 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45763.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43920.log 2024-09-10T20:33:27.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39977.log.gz 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50026.log 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45763.log.gz 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55108.log 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48808.log 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50026.log.gz 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54142.log 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55108.log.gz 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38963.log 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48808.log.gz 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66078.log 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54142.log.gz 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45362.log 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38963.log.gz 2024-09-10T20:33:27.296 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.66078.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57250.log 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66078.log.gz 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63051.log 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45362.log.gz 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66540.log 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57250.log.gz 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69480.log 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63051.log.gz 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51118.log 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66540.log.gz 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46051.log 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69480.log.gz 2024-09-10T20:33:27.297 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62085.log 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51118.log.gz 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46051.log.gz 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56956.log 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62085.log.gz 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46127.log 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50950.log 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56956.log.gz 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57418.log 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46127.log.gz 2024-09-10T20:33:27.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37371.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50950.log.gz 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62547.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57418.log.gz 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67842.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37371.log.gz 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60783.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62547.log.gz 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68430.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67842.log.gz 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66036.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60783.log.gz 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68430.log.gz 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64943.log 2024-09-10T20:33:27.299 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66036.log.gz 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68262.log 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62799.log 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64943.log.gz 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58132.log 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68262.log.gz 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66708.log 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62799.log.gz 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63303.log 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58132.log.gz 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42928.log 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66708.log.gz 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.63303.log: /var/log/ceph/ceph-client.admin.51202.log 2024-09-10T20:33:27.300 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63303.log.gz 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.42928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46720.log 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42928.log.gz 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51454.log 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51202.log.gz 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62715.log 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46720.log.gz 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51454.log.gz 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48850.log 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62715.log.gz 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52378.log 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60741.log 2024-09-10T20:33:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48850.log.gz 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40963.log 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52378.log.gz 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33642.log 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60741.log.gz 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40963.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40963.log.gz 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60909.log 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33642.log.gz 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66456.log 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43267.log 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60909.log.gz 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70532.log 0.0% 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66456.log.gz 2024-09-10T20:33:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65321.log 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43267.log.gz 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70532.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.70532.log.gz 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/ceph-client.admin.63261.log 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38043.log 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65321.log.gz 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63261.log.gz 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69018.log 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65364.log 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38043.log.gz 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69018.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69018.log.gz -5 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54100.log 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64355.log 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65364.log.gz 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54100.log.gz 2024-09-10T20:33:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69943.log 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64355.log: 96.2% -- replaced with /var/log/ceph/ceph-client.admin.43920.log.gz 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50656.log 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64355.log.gz 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53176.log 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69943.log.gz 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70280.log 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50656.log.gz 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53176.log.gz 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43181.log 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70280.log.gz 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40347.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz --verbose 2024-09-10T20:33:27.304 INFO:teuthology.orchestra.run.smithi103.stderr: -- /var/log/ceph/ceph-client.admin.63472.log 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40421.log 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43181.log.gz 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63472.log.gz 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67212.log 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49984.log 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr: 95.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40421.log.gz 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67212.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.67212.log.gz --verbose 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr: -- /var/log/ceph/ceph-client.admin.40754.log 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37287.log 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49984.log.gz 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51538.log 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-09-10T20:33:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64229.log 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37287.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67422.log 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51538.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64229.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50404.log 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40791.log 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67422.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37769.log 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50404.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52882.log 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40791.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53302.log 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37769.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52882.log.gz 2024-09-10T20:33:27.306 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38211.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66162.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53302.log.gz 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38211.log.gz 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64691.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66162.log.gz 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59520.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64691.log.gz 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67884.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58342.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59520.log.gz 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61833.log 2024-09-10T20:33:27.307 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67884.log.gz 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.58342.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45977.log 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58342.log.gz 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57838.log 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61833.log.gz 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38085.log 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45977.log.gz 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38397.log 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57838.log.gz 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44440.log 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38085.log.gz 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54688.log 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38397.log.gz 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56704.log 2024-09-10T20:33:27.308 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44440.log.gz 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33729.log 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54688.log.gz 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56704.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56704.log.gz -5 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43810.log 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50992.log 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33729.log.gz 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67044.log 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43810.log.gz 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67338.log 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50992.log.gz 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38852.log 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67044.log.gz 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69564.log 2024-09-10T20:33:27.309 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67338.log.gz 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69522.log 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38852.log.gz 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65784.log 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69564.log.gz 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40014.log 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69522.log.gz 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58216.log 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65784.log.gz 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56536.log 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40014.log.gz 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58216.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.58216.log.gz -- 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/ceph-client.admin.52504.log 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43559.log 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56536.log.gz 2024-09-10T20:33:27.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51706.log 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52504.log.gz 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50278.log 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43559.log.gz 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51706.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51706.log.gz -5 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69648.log 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62211.log 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50278.log.gz 2024-09-10T20:33:27.311 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56746.log 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69648.log.gz 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56200.log 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62211.log.gz 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55696.log 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56746.log.gz 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64523.log 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56200.log.gz 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51790.log 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55696.log.gz 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55024.log 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64523.log.gz 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55528.log 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51790.log.gz 2024-09-10T20:33:27.312 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64439.log 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55024.log.gz 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53638.log 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55528.log.gz 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67632.log 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64439.log.gz 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64145.log 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53638.log.gz 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43431.log 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67632.log.gz 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57754.log 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64145.log.gz 2024-09-10T20:33:27.313 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67296.log 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gz 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39222.log 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57754.log.gz 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49102.log 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67296.log.gz 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52588.log 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39222.log.gz 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65868.log 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49102.log.gz 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48724.log 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52588.log.gz 2024-09-10T20:33:27.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49564.log 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65868.log.gz 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50866.log 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48724.log.gz 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.49564.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49564.log.gz 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71456.log 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50866.log.gz 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71288.log 2024-09-10T20:33:27.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71456.log.gz 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56242.log 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71288.log.gz 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50362.log 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56242.log.gz 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50320.log 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62883.log 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50362.log.gz 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44356.log 2024-09-10T20:33:27.316 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50320.log.gz 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49480.log 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62883.log.gz 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59394.log 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44356.log.gz 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65616.log 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49480.log.gz 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59394.log.gz 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56452.log 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51664.log 2024-09-10T20:33:27.317 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65616.log.gz 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70868.log 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56452.log.gz 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51664.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51664.log.gz 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70868.log.gz 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68850.log 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60657.log 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67506.log 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68850.log.gz 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58638.log 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60657.log.gz 2024-09-10T20:33:27.318 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62925.log 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67506.log.gz 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58806.log 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58638.log.gz 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69774.log 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62925.log.gz 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51580.log 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58806.log.gz 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68934.log 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69774.log.gz 2024-09-10T20:33:27.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46609.log 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51580.log.gz 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71120.log 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68934.log.gz 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56326.log 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46609.log.gz 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49606.log 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71120.log.gz 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55654.log 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56326.log.gz 2024-09-10T20:33:27.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51916.log 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49606.log.gz 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55654.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.55654.log.gz /var/log/ceph/ceph-client.admin.61287.log 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51916.log.gz 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.61287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66330.log 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61287.log.gz 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33444.log 2024-09-10T20:33:27.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62043.log 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66330.log.gz 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33444.log.gz 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39370.log 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69312.log 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62043.log.gz 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44062.log 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39370.log.gz 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39296.log 2024-09-10T20:33:27.322 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69312.log.gz 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60699.log 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44062.log.gz 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45196.log 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39296.log.gz 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44860.log 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60699.log.gz 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45196.log.gz 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55192.log 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58722.log 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44860.log.gz 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56578.log 2024-09-10T20:33:27.323 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55192.log.gz 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.58722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65111.log 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58722.log.gz 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56578.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.56578.log.gz 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69228.log 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65237.log 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65111.log.gz 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48682.log 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69228.log.gz 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38815.log 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65237.log.gz 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64901.log 2024-09-10T20:33:27.324 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48682.log.gz 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63935.log 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38815.log.gz 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52630.log 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64901.log.gz 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57922.log 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63935.log.gz 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68136.log 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52630.log.gz 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67758.log 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57922.log.gz 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48934.log 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68136.log.gz 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69102.log 2024-09-10T20:33:27.325 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67758.log.gz 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40865.log 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48934.log.gz 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55864.log 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69102.log.gz 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57628.log 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40865.log.gz 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55150.log 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55864.log.gz 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.57628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68682.log 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57628.log.gz 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44104.log 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55150.log.gz 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54310.log 2024-09-10T20:33:27.326 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68682.log.gz 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63177.log 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44104.log.gz 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69270.log 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54310.log.gz 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40680.log 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63177.log.gz 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50068.log 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69270.log.gz 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40680.log.gz 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54814.log 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48766.log 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50068.log.gz 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45829.log 2024-09-10T20:33:27.327 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54814.log.gz 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62841.log 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48766.log.gz 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42929.log 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45829.log.gz 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46646.log 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62841.log.gz 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.42929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37187.log 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42929.log.gz 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43351.log 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46646.log.gz 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.37187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48514.log 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37187.log.gz 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68472.log 2024-09-10T20:33:27.328 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43351.log.gz 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48514.log.gz 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53806.log 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66414.log 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68472.log.gz 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40606.log 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53806.log.gz 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.66414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70364.log 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66414.log.gz 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67002.log 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40606.log.gz 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64817.log 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70364.log.gz 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50698.log 2024-09-10T20:33:27.329 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67002.log.gz 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54646.log 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64817.log.gz 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62295.log 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50698.log.gz 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53344.log 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54646.log.gz 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59646.log 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62295.log.gz 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59730.log 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53344.log.gz 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59940.log 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59646.log.gz 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62337.log 2024-09-10T20:33:27.330 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59730.log.gz 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39407.log 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59940.log.gz 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51034.log 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62337.log.gz 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71204.log 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39407.log.gz 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39518.log 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51034.log.gz 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.71204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55822.log 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71204.log.gz 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39481.log 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39518.log.gz 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-09-10T20:33:27.331 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55822.log.gz 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.39481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46498.log 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39481.log.gz 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64019.log 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46498.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46498.log.gz 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38564.log 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64019.log.gz 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56662.log 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.38564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65574.log 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38564.log.gz 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69396.log 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56662.log.gz 2024-09-10T20:33:27.332 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52924.log 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65574.log.gz 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.69396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40125.log 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69396.log.gz 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68010.log 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52924.log.gz 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53260.log 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40125.log.gz 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.68010.log: -- /var/log/ceph/ceph-client.admin.48556.log 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68010.log.gz 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.53260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68766.log 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53260.log.gz 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.48556.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60444.log 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48556.log.gz 2024-09-10T20:33:27.333 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68766.log.gz 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.60444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67590.log 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60444.log.gz 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64565.log 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.67590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68304.log 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67590.log.gz 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64565.log.gzgzip 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56998.log 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59898.log 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.68304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68304.log.gz 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45070.log 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56998.log.gz 2024-09-10T20:33:27.334 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.59898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43645.log 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59898.log.gz 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54436.log 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45070.log.gz 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43645.log: /var/log/ceph/ceph-client.admin.65700.log 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43645.log.gz 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.54436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43771.log 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54436.log.gz 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50782.log 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.65700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65700.log.gz 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.43771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48304.log 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43771.log.gz 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.50782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57964.log 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50782.log.gz 2024-09-10T20:33:27.335 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.48304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38480.log 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48304.log.gz 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.57964.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45703.log 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57964.log.gz 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.38480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33091.log 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38480.log.gz 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62463.log 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.45703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45703.log.gz 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-client.admin.33091.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56494.log 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33091.log.gz 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52462.log 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62463.log.gz 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.56494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40199.log 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56494.log.gz 2024-09-10T20:33:27.336 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.52462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52462.log.gz 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62127.log 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.40199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70238.log 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40199.log.gz 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49312.log 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62127.log.gz 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.70238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70238.log.gz 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49228.log 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49312.log.gz 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.44398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63430.log 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.49228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64313.log 2024-09-10T20:33:27.337 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49228.log.gz 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.63430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51286.log 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63430.log.gz 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.64313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62757.log 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64313.log.gz 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.51286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55486.log 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51286.log.gz 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.62757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62757.log.gz 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46424.log 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55906.log 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55486.log.gz 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.46424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46424.log.gz 2024-09-10T20:33:27.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.55906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55906.log.gz 2024-09-10T20:33:28.284 INFO:teuthology.orchestra.run.smithi103.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-10T20:33:28.926 INFO:teuthology.orchestra.run.smithi103.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-10T20:33:29.691 INFO:teuthology.orchestra.run.smithi103.stderr: 89.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-10T20:33:30.669 INFO:teuthology.orchestra.run.smithi103.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-10T20:40:07.936 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-10T20:42:32.246 INFO:teuthology.orchestra.run.smithi103.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-10T20:43:26.384 INFO:teuthology.orchestra.run.smithi103.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi103.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-10T20:44:09.014 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-10T20:44:09.015 INFO:teuthology.orchestra.run.smithi103.stderr:real 10m42.073s 2024-09-10T20:44:09.015 INFO:teuthology.orchestra.run.smithi103.stderr:user 5m44.312s 2024-09-10T20:44:09.015 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m14.402s 2024-09-10T20:44:09.015 INFO:tasks.ceph:Archiving logs... 2024-09-10T20:44:09.015 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553/remote/smithi103/log 2024-09-10T20:44:09.016 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T20:45:10.157 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T20:45:10.167 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T20:45:10.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T20:45:10.240 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T20:45:10.248 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T20:45:10.248 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:45:10.263 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:45:10.295 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:45:10.295 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-10T20:45:10.295 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 7 -648us[ -639us] +/- 51ms 2024-09-10T20:45:10.295 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 7 +1957us[+1966us] +/- 67ms 2024-09-10T20:45:10.295 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -1566us[-1566us] +/- 61ms 2024-09-10T20:45:10.295 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:45:10.296 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T20:45:10.304 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T20:45:10.304 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T20:45:10.313 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T20:45:10.459 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:45:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:45:10.488 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T20:45:10.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T20:45:10.543 INFO:teuthology.task.internal:Duration was 1979.557379 seconds 2024-09-10T20:45:10.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T20:45:10.579 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T20:45:10.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T20:45:10.676 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:45:11.012 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T20:45:11.012 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-09-10T20:45:11.013 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T20:45:11.034 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T20:45:11.034 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T20:45:11.101 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T20:45:11.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T20:45:11.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T20:45:11.310 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T20:45:11.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T20:45:11.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T20:45:11.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:45:11.436 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-09-10T20:45:11.479 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:45:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:45:11.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T20:45:11.543 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T20:45:11.544 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899553/remote/smithi103 2024-09-10T20:45:11.545 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T20:45:11.609 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T20:45:11.610 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T20:45:11.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T20:45:11.672 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T20:45:11.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T20:45:11.683 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T20:45:11.683 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T20:45:11.714 INFO:teuthology.orchestra.run.smithi103.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 20:45 /home/ubuntu/cephtest 2024-09-10T20:45:11.716 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T20:45:11.726 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T20:45:11.854 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest}} duration: 1979.5573785305023 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-10T20:45:11.855 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:45:11.957 INFO:teuthology.run:pass